As I’ve suggested in recent posts I’m finding it quite challenging to make sense of this roiling debate about AI safety. When I write about American politics, American history and a few other topics I have a body of knowledge that I draw on and try to share with readers — explain how it might illuminate present-day questions. I may be right or wrong. But I’m speaking from a background of some real knowledge and study. When I delve into questions like AI or AI dangers I’m seldom trying to come up with definitive answers. I lack the qualifications or foundational knowledge. What I’m trying to do is first try to educate myself, but in my writing I at least try to frame the basic questions, provide a framework for considering the issue, point to what are at least good faith and knowledgable voices if not necessarily people who are “right.” But while I’m still uncertain about dangers posed by the companies doing the most advanced and often reckless research on artificial intelligence I’ve been increasingly stuck on the prominence in the AI world of people who are in the overlapping communities of “effective altruism,” “rationalism” and “longtermism”.
Just to point to the latest examples, the Journal has a write-up on Jacob Coxon, the AI researcher whose Twitter thread quite literally triggered the current freakout/reckoning. It turns out his formative years were spent in the various parts of the “rationalist” subculture in London before he migrated to the U.S.
I’ve been loosely aware of the first and the second for a while, but didn’t have much pressing need to learn more because they didn’t seem connected to much that I care about. But they’re very big in the AI world and maybe even bigger in the AI will wipe out humanity worlds. So given the global freakout getting a handle on this becomes more pressing. Just as an aside, I will note that I don’t know how the dorks operating under the “rationalism” label secured exclusive branding rights to that term. Like, rough day for Descartes and quite a few other philosophers. But I digress. What I’m going to argue here is that at least the popularized versions of what I’ll call ERL (“effective altruism”, “rationalism” and “longtermism”) is made up of a mix of absurd and nonsensical ideas. The fact that so many AI doomers are part of that world gives me a lot of pause. But we can’t go from ERL is absurd to AI is fine.
So let’s get down to it.
At its simplest and least objectionable, effective altruism holds that we should make decisions about philanthropy and the work we do in life not on the basis of emotion or sentiment but rigorous and fact-based analysis of which actions accomplish the most good. This is good advice and almost impossible to disagree with. Indeed, many people try to do this who don’t have any label but just try to make good decisions. The key is where logic and rationalism come in. As we discussed in an earlier post the “rationalists” are very focused on future extinction events and how to avoid them. And often when we say future we mean really far in the future. Like centuries or millennia or even millions of years in the future. In 2017 a Scottish philosopher named William MacAskill first used the term “longtermism” within the effective altruism movement. This was essentially giving a name to this focus on the often distant future and extinction events. (Here’s a 2022 MacAskill essay on what the terms means.)
The essence of “longtermism” is using posited future extinction events often in the very distant future as the basis of moral reasoning about the present. There’s also some alluring reasoning about numbers of people. There are 8 billion of us today. But if you consider the probable lifetime of humanity, which we’re told is around 750,000 or so years, there might be trillions of future humans whose lives are on the line on the basis of actions we take today. And what are the claims of, say, a billion of today’s humans versus the trillions of future humans because of this or that extinction event?
A central critique of this thinking is that it often leads to ethical arguments for actions that are somewhere on the spectrum between problematic and evil. That is a very important critique. I’ve struggled to get past a more elemental one. Most of these future events are little more than made up science fiction dressed up in the language of rationalism about which we know absolutely nothing. As I put it on Bluesky today, where I confess I write in a less restrained manner, “basically a movement or subculture of morons who propose novel forms of moral reasoning based on sci-fi ideas they imagine might happen hundreds, thousands or millions of years in the future.”
Perhaps not coincidentally, I first remember hearing the term “longtermism” around 2022 or 2023 when Grimes, the singer-songwriter and estranged former partner of Elon Musk commented on his apparent turn to the hard right. She said she’d never seen that in him. They just connected on mutual interest in things like “longetermism.”
In any case, humans have little ability to see more than perhaps fifty years into the future. This is a difficult argument to make theoretically or perhaps scientifically. But reading over the thoughts of previous generations, it has a lot of basis empirically. It’s a time horizon over which we have some ability to see the outlines of future technologies, artistic and cultural movements, demographic changes and more. Needless to say, literally predicting the future is completely impossible. But we can see some of the outlines over this time horizon. We have zero idea about anything 1,000 years from now. We know essentially nothing about anything 150 years from now, tens of the thousands of years from … you get the idea.
As an aside, from the Journal article I learned that it was Dominic Cummings, a key architect of Brexit who brought the “rationalist” milieu Coxon was part of into UK politics. So much for seeing much into the future.
We should further not say that those trillions of people in the future do not exist. At all. One might as well get into a conversation about the offspring you and your spouse condemned to an eternity of non-being because you didn’t have sex on a particular night, perhaps at a particular minute.
I’m not trying to get into weird, modern versions of arguments about the number of angels who can fit on the head of a pin. Indeed, I’m trying to make sure we don’t inadvertently fall into them. Most of these ideas, most of these proposed modes of moral reasoning are based on a wildly grandiose and hubristic confidence in our ability to know the future or even make exacting estimations of the time in which we ourselves live. Whatever lessons you draw or moral reasonings you make based on made up things in the distant future has to be meaningless. Because you know nothing about what is going to happen in the distant future. But that is kind of the coin of the realm in this somewhat culty ERL subculture.
As I said, these people’s weird or even suspect ideas don’t negate fears about dangers created by AI. A lot of these people are not only into “rationalism” and “longtermism,” but they’re also the ones who’ve done a lot of work on building AI. That must give them some credibility when it comes to discussing its possible dangers. But a decent amount of the skepticism about AI doomerism is based on this backstory. And even though it doesn’t mean we should discount claims about AI and existential risk we should at least factor into the equation the world view of a significant number of people making these claims. It’s not the easiest balance to draw. Because this is a version intense and polarized debate. But we should try.