You’ve probably seen these reports of an employee of the AI company Anthropic who quit his job and issued a warning on X that suggested AI could, in next few years, escape control and lead to the extinction of humanity. These sound like not just wild but science fiction-type claims. And they’ve lurked around frontier AI labs for years. It’s one of the many weirdnesses of the AI world and those who lead it. The basic pitch: AI is awesome. We need to built it as quickly as possible. Also it might lead to human extinction. Quite an attractive offer! There’s always been a strong sense among many observers that these claims or suggestions or warnings are part of the AI hype machine itself, albeit of a kind of contradictory or paradoxical variety.
But this warning by the ex-Anthropic employee, Jacob Coxon, seems different and is unquestionably being reacted to very differently. That’s the one part of this that is new and real – the reaction to this warning/comment etc is much bigger and operating in the tech, financial and general media. Wired has a good interview with him here. (I think you can read it as a free article if you haven’t read other Wired articles this month.) The gist is that Coxon says it’s imperative to create a regulatory structure, or at least an ad-hoc agreement that can slow the competition between OpenAI and Anthropic, the two most advanced AI engines and research entities. (The inflection point people are focusing on is something called “recursive self-improvement,” which is when this generation of AI model builds the next one.) The problem is that you really need an agreement that brings China into some common framework too. Because they’re in this same hunt, running these same risks, even though I think the common consensus is that Chinese companies are running at least somewhat behind the most advanced U.S. companies. China’s strength has been building models which are only a bit behind the U.S. models but at dramatically lowest costs.
This all leads to a pretty dismal place, even on top of any versions of these warnings and threats being credible. We have a gung-ho White House which is full speed ahead on everything AI, has been highly skeptical of Anthropic’s warnings on this front (which they’ve characterized as a business ploy), is obsessed with beating China on AI, and, as much as all that, lacks the competence and attention span to manage anything remotely like this. It’s an understatement that the idea of the Trump White House leading any kind of international agreement to control or manage the advance of AI is almost beyond laughable.
If any of this is true, there’s no one at the national driver’s wheel for a good two and a half years, and at least what these guys are saying is that that same general time frame is the window of real danger.
Like most of us, I have no ability on my own to evaluate any of this. What does seem true though is that this isn’t just marketing hype or something generated out of the limitless grandiosity of the biggest tech oligarchs. A lot of people developing the technology are seriously concerned about this. Let me note a couple additional points on which I’m no expert but have some grounded ideas.
(Again, if you can access the Wired interview above I recommend it. It does a fairly good job reducing to concrete terms what on its face sounds fantastical.)
The first point is that there’s a separate very wild and woolly conversation in Silicon Valley about whether AI is, might soon be, or will ever be sentient or self-aware. In general, the people from Silicon Valley or the AI labs who talk up this possibility usually seem to be operating on a very juvenile idea of what sentience or self-awareness even is. So when you actually hear them get beneath the headlines at least my response has been something like, “I can’t take anything you’re saying seriously because you don’t seem to have any serious understanding of what sentience even is.” Basically, just tech bros yakking.
What we’re talking about here has really nothing to do with that. The question is whether AI scientists can control what they create, or predict what the things they create will do. And we’ve already seen a few small-scale but real examples that they can’t.
The other thing is that a very big part of the U.S. and the global economy is now heavily leveraged on AI. I wrote earlier this morning about how AI’s hunger for capital is already driving up the costs of government borrowing around the world. In this case, though, we have built hundreds of billions, perhaps trillions of dollars of incentives against slowing any of this development down. That’s really bad, and suggests massive obstacles to any kind of preventive action. Indeed, it’s not crazy to think that the kind of pause or moratorium or even slowdown of AI development Coxon says should happen would trigger a correction in AI valuations that could drive the economy into a recession. To the extent the macro economy is doing well right now and people are making a lot of money in the equities market, even if it’s overwhelmingly going to the wealthiest people, it’s heavily based on bets and spending on AI.
In any case, these are scary warnings, and I think we should take them seriously. Even if the probabilities and scope of danger is much less than these folks are saying we’re talking about very, very bad outcomes. But when you look outside the technology itself to the current political economy and state of global governance, it gets scarier still.