Inside a Perfect Storm — How Dangerous Is AI?

You’ve probably seen these reports of an employee of the AI company Anthropic who quit his job and issued a warning on X that suggested AI could, in next few years, escape control and lead to the extinction of humanity. These sound like not just wild but science fiction-type claims. And they’ve lurked around frontier AI labs for years. It’s one of the many weirdnesses of the AI world and those who lead it. The basic pitch: AI is awesome. We need to built it as quickly as possible. Also it might lead to human extinction. Quite an attractive offer! There’s always been a strong sense among many observers that these claims or suggestions or warnings are part of the AI hype machine itself, albeit of a kind of contradictory or paradoxical variety.

But this warning by the ex-Anthropic employee, Jacob Coxon, seems different and is unquestionably being reacted to very differently. That’s the one part of this that is new and real – the reaction to this warning/comment etc is much bigger and operating in the tech, financial and general media. Wired has a good interview with him here. (I think you can read it as a free article if you haven’t read other Wired articles this month.) The gist is that Coxon says it’s imperative to create a regulatory structure, or at least an ad-hoc agreement that can slow the competition between OpenAI and Anthropic, the two most advanced AI engines and research entities. (The inflection point people are focusing on is something called “recursive self-improvement,” which is when this generation of AI model builds the next one.) The problem is that you really need an agreement that brings China into some common framework too. Because they’re in this same hunt, running these same risks, even though I think the common consensus is that Chinese companies are running at least somewhat behind the most advanced U.S. companies. China’s strength has been building models which are only a bit behind the U.S. models but at dramatically lowest costs.