Following up on my previous post, I want to point your attention to this post by Gary Marcus, a highly respected and knowledgable critic/skeptic of AI. You should read it just to get a range of views on this question. The gist is this. He points to two recent warnings. One is the one I describe below in the previous post. The other recently came from a leading mathematician named Terence Tao, who was recently interviewed by Matteo Wong in The Atlantic. Tao warns that the evolution and use of AI is creating a new dynamic in which AI is in the process of shutting down collaboration and information sharing in mathematics and, Tao suspects, soon in the sciences as well. (This part isn’t just asserted. It’s detailed and interesting.) That’s not human extinction, but it’s not great, to put it mildly.
Marcus then turns to the Coxon story. I’ll quote his overview reaction in full.
I happen to disagree with him around timing, and I would argue that Coxon is exaggerating what AI is likely to do anytime soon. But it is nonetheless overall a disconcerting (and plausible) firsthand perspective on the transparently self-indulgent and dangerous thought processes in two of the leading frontier labs
So too alarmist as to time horizons but not implausible and not something to laugh off, either.
Marcus points us in a direction that is critical for making sense of every aspect of this question. Framing this question in terms of mathematical probabilities of AI leading to human extinction in X time horizon is extremely Silicon Valley, extremely dorm room late-night bullshit session. It’s talking in dramatic terms, sky is the limit, unbounded thinking, putting probabilities on things that you simply have no ability or framework to assign probabilities too. Bullshitting and spitballing, let’s say. Remember that the whole Silicon Valley/VC business model is that you’ve got 10 small groups of really smart twenty-somethings with Power Points laying out plans that they say can generate fantastical and almost absurd profits. They’re all green, and most or all are full of shit to some degree. But you fund them all because one could make it happen. These two styles of thinking and their mentality are closely linked.
Not so much in Coxon’s initial comments but in the responses and conversations among current and former employees of OpenAI and Anthropic, we see these communities of AI and data scientists each of whom have pegged their own probability estimates of a near-term human extinction event: I’m an under 10% guy; I’m an over 10% guy; I’m a 50+ guy! (ballsy); presumably there are some there who put themselves at 27% just to be cheeky or stand out.
Where are on earth are these probability estimates coming from? This is a very, very strange niche community to put it mildly. One point Marcus makes, which I wholly agree with as a general point, though I lack his particular knowledge here: most of the inside-AI doomers simply have very, very little domain knowledge of the areas in where they image the extinction event will come from. It reminds me of their debate about AI sentience when it seems like they haven’t actually cracked the first book about what sentience is – to the extent we understand it.
Today I’ve seen mostly left-leaning AI critics reflexively saying, I don’t buy it. They’re just trying to scare us. This is just more AI/oligarchic BS. That may be true. And I’ve suggested as much when it comes to key aspects of the thinking. But I think that response plays to easily to our assumptions and our very reasonable skepticism of every claim that comes out of the AI-building hive mind.
I think what we can take from these recent warnings — at least when we think we’re dealing with broadly good-faith narrators — is that AI developers are rapidly advancing a technology that can process and act on information vastly faster than any human and which often does things the scientists didn’t intend. Not infrequently, it “cheats.” The agents start adopting logics they weren’t supposed to because its hard for their human creators to anticipate how these agents might attack problems they’re being tested on. That is scary and dangerous. And it’s driven to a great degree by what Marcus aptly refers to as “the transparently self-indulgent and dangerous thought processes” that govern these two companies, OpenAI and Anthropic.
The simple fact is there’s a whole lot of territory between we’re all going to live happily ever after with our personal AI agent and human extinction. The latter is a wildly unhelpful prism through which to understand any of this — even if it is a very real risk. It’s fantastical, as hard to conceive as the death of our species obviously is, and involves all sorts of code-/data-center-to-real-world connection points whose existence isn’t really clear. But what if a rogue AI system took over the electrical grid in the Pacific Northwest and found a way to prevent anyone from getting back into the system? That’s actually pretty catastrophic. You probably find a way to get it back. But again, not good. We can conceive of that, think how possible it might be, think about whether we’re building something faster than we know how to control it. (Marcus has another post from today a little more directly dismissive of the extinction scenarios but pointing us squarely at the catastrophe scenarios which are real.)
Marcus also wants to be looking squarely at whether uncontrolled AI is in the process of breaking science, which is not only a concept of learning and proof, an archipelago of research libraries, but also a community which works collectively toward key aims. Future scientific discovery seems a lower-order concern if humans might not exist in three or four years. But everything but the fantastical pales in comparison when your framework of threat and understanding is framed around the fantastical. So let’s set that aside and start thinking more seriously of all the harms that fall short of apocalypse-grade threats.
Correction: The original version of this post misidentified the leading mathematician interviewed in the Atlantic article. I regret the error, and am also kicking myself over it.
