For years, and especially in the last month or so, we’ve been collectively inundated with an AI industry which, uncannily and bizarrely, tells us simultaneously that AI is essential, inevitable and runs a significant chance of exterminating all of us in as few as two or three years. If you’ve dipped into this debate at all, you’ve quickly come upon discussions of so-called “rationalists” or “long-termists” or supporters of “effective altruism” who are simultaneously ubiquitous among those developing AI and in the AI safety (i.e., the folks sounding the alarm) community as well. (For the sake of simplicity and brevity, I’ll refer to these folks as the “REL nexus”: rationalists, effective altruists and long-termists. This is my coinage since I don’t know a better one.) I’ve spent a fair amount of time familiarizing myself with this world and how it relates to current debates about AI. So I wanted to share my basic impressions.
My global impression is that REL thinking really does suffuse the whole Silicon Valley and Silicon Valley-adjacent AI world, and that the risk projections about the chances of AI-driven extinction or existential risk are part of a belief system and set of certainties that long predates the entire existence of Large Language Models, which are what public AI discourse is really about. (By that, I mean there are other forms of machine learning under the AI heading which predate LLMs and other forms in development. But everything we’re currently talking about in the public sphere are LLMs.) There are both really fantastical (not necessarily false but fantastical) beliefs about AI’s potential and also its dangers. For me, it’s a big deal that this tightly woven set of beliefs predates almost all the empirical realities we’re seeing and dealing with today.
Something related applies to what is usually referred to as AGI (artificial general intelligence). This is a major part of the AI debate, and key parts of the public debate center on how close AGI may be. But there’s no real clarity about what AGI even is. Broadly, what it means is AI with a general intelligence — i.e., it has intelligence across most or all of the domains in which humans operate and is superior to that of humans. Not just language, not just number crunching, but most or all of them. Because of this, it becomes significantly autonomous. You don’t have to train it on new things or add new programming. It can learn things itself somewhat in the way that we can learn things. If you or I set our minds to it we could develop expertise in geography, mindfulness, plane-flying, mathematics, providing psychotherapeutic counseling. AGI could do all those and most of the other things humans can and do them better — with more effective and powerful reasoning, faster etc.
But as I noted above, this is a posited concept that long predates the AI/LLM breakthroughs of the last half dozen years or so. It’s a very ambiguous idea with very few hard parameters for what constitutes it or how we’d know it’s been developed. It seems relatively untethered to our more general understandings of what intelligence, or at least intelligence in humans even is. And it’s unclear that we’re close to it whatever it might be. Again, we’re talking about a tightly woven set of beliefs that predate all the current empirical findings about what LLMs are, how they work, etc.
Now, some people are just really good at seeing over the horizon. It was very clear to physicists early on that Relativity Theory pointed to the possibility of releasing unimaginable amounts of energy by splitting atoms and engineering uncontrolled chain reactions. They could also see the implications and dangers associated with creating technologies that could do that in a controlled and scalable way. They were right. It was pretty bad, a genie that could never really be placed back in its bottle, and we’ve been living with that reality since 1945. But I don’t think this is like that. The role of empiricism, the scientific method, and falsifiability are just fundamentally different. What we’re talking about is more like a culture, an interwoven set of beliefs, that have only an uncertain relationship to the technologies (LLMs) that we’re talking about. To an extent, we’re also talking about a set of beliefs that is highly centered within the community and mentality of engineering with too little grappling with what takes place outside of that world.
If you read up on people in the REL world (again, my coinage, don’t Google REL) you’ll hear that it’s adjacent to a lot of right-wing thinking, a lot of eugenicist thinking, a lot of very suspect moral reasoning that involves doing dark or evil things in the present in the interests of large numbers of theoretical future people who REL folks think will exist 10,000 years from now. What is the moral standing or worth of a mere billion people today compared to a trillion people who might be alive in a million years across the galaxy? Or the trillion that that might live over the next 10,000 years? (No, I’m not kidding. Don’t get me started.)
Each of these claims are true to some degree. But the bigger point is that people in the REL world are far too loose with their beliefs in things that we have no evidence are true. They posit technologies with no clear path to making them and engage in a way of thinking about probability that frankly just isn’t how the world actually works. In a very broad sense, the whole community talks a lot with the language of science without the practice science, by which I mean frameworks of proof, falsifiability and specifics. And just to bring this back to specifics, why should any of us care what these people in what we’re short-handing as the REL community think? Because the great majority of what you hear about AI safety in the context of extinction events and existential risks comes back to them or is highly shaped by their thinking. A lot of things that are discussed in the REL world as real, tangible things, albeit things that are still a bit off in the future, at best stretch the limit between reality and science fiction. When I was kid and significantly into adult life and to a significant degree still today I too am really into Star Trek and thinking about what things could be real and what the limits of our imagination are. But that’s not a valid basis for making public policy. And it’s mostly not the basis of ethical judgments in the here and now or on planet Earth.
Now, these are some pretty big claims I made above, especially coming from someone who struggled with high school math. I can only say that I try to be very aware of what I don’t now, areas of knowledge that are mostly unknown to me, attentive to how people do and don’t engage with genuine domain knowledge, how people reason in areas I do know something about. (When I hear top AI executives trying to convince clergy that LLMs are in some sense conscious and have moral standing akin to humans, it makes me think I’m not dealing with serious people.) The credentialed drivers of the REL community tend to be not in the sciences but rather in academic philosophy. And the more I’ve learned, the more I’ve been genuinely baffled that people who think in these ways have managed to secure quite prestigious academic appointments. In any case, I’m not asking you to take my word for any of this stuff. I’m giving you my perspective on a question at the center of a critical public conversation.
Now, where does this leave us? I want to be clear that I’m emphatically not saying AI is awesome and let’s go full speed ahead. Just because this whole conversation about AI exterminating humanity comes from a very iffy place doesn’t mean that AI is therefore safe. That’s a basic fallacy. It doesn’t even necessarily mean it doesn’t pose existential risks. It’s just that the current claims are too infected, too much fruit of the poisoned or nonsensical, quasi-religious tree to draw any from them. And this has been where I’ve been kind of hung up. There are clearly a lot of things we need to worry about and take steps to deal with about AI right now. One of those is the fact that a lot of AI development and testing has been quite reckless, taking risks and assuming society will absorb the consequences, using either sloppy or inexpert security precautions to wall off testing (in which frontier labs are often testing whether their models will do damaging things) from the larger web. To the extent that there are near term big risks from current LLMs, they seem much more likely to come from bad actors using powerful AI to do bad things than LLMs “deciding” to do bad things because their actions are unpredictable.
What I think we as citizens and as people involved in the civic process need to do is focus on concrete potential dangers that have evidence behind them, specific arguments and to-dos. And we proceed on that basis. We shouldn’t be starting from claims about the potential of LLMs exterminating humanity and reasoning back to slightly less worse things and thinking of how to stop them. One of the most consistent patterns of the REL world is to posit maximal risks or opportunities, assign degrees of risk to them, often on the basis of no more than guesswork or intuition, and then start building whole structures of ideas and plans downstream of those initial postulates or assumptions. But often the assumptions — the scientific-sounding, numerical percentages of risk — are really based on nothing.
On Monday, Politico published an interview from what appears to be OpenAI CEO Sam Altman’s most recent post-controversy media tour in which he said, “We believe that the world should accept some bad things happening for the benefits of this technology and people having the agency.” As one person pointed out to me, on its face, or out of context, this isn’t necessarily such a controversial remark. We accept tens of thousands of people dying every year from auto accidents as the price of having cars. Those are definitely “bad things.” Every major technology has some risks inherent in it. But in the context of AI and the current public conversation about AI risks and downsides, this comment epitomizes the mentality behind Silicon Valley’s rush to embrace AI. “Bad things” covers a pretty expansive amount of territory. Altman and his team think we should accept whatever “bad things” means because AI is just so amazing and will bring so many wonderful things. But just what those wonderful things are seems highly speculative and uncertain. And big majorities of Americans are at best uncertain whether they want them at all. What you get from the collision of these two realities is that yet again it’s Silicon Valley’s world and we’re just living in it. That’s fundamentally what the data center backlash is about. It’s the one piece of the equation where ordinary people — because of these specific details of zoning law — can say no. We should see this wild and ecstatic boosterism — that AI is our salvation and destiny — as just the flip side of the doomerism that predicts our near-term extinction. They both flow downstream from a series of assumptions and beliefs, which when you see them close up, are genuinely weird, usually unsupported by actual evidence and often based on ethical assumptions and values that are simply bad.