This is an update to my post from Tuesday about the “existential risk” panic. A good friend of mine reached out to me and said, in so many words, yes, all that techno-babble is weird. But that doesn’t mean these risks aren’t real. And as I tried to make clear in my piece, I agree with that, at least to the extent that if Group A isn’t made up of reliable narrators, I’m not clear who the reliable narrators are. He pointed me to this recent report put out by the Rand Corporation, which is kind of the original think tank and has always been closely tied to the U.S. government and for lack of a better word the U.S. military-intelligence-industrial complex. It’s an interesting report. Or at least I found some interesting and I think sensible prescriptions in it once I got through the first half or two-thirds of it, which contained very Randian decision trees, definitions and a lot of what seemed like assumptions about existential risk and coexistence with AI that I didn’t understand. It eventually sets out a sensible if hard-to-execute approach which generally amounts to a safety-first regime of governing AI (make sure new things don’t endanger us before we build them or make them widely available) and, critically, open the decision-making up to public institutions with public accountability.
This is sensible and obvious. And it’s so obvious and so necessary that it’s a little disturbing that it’s even up for debate. We don’t let munitions makers just fiddle around and experiment with no oversight or limits on who they sell things to. It was obvious from the git-go when we were developing nuclear weapons and nuclear power that we wouldn’t leave that stuff to just wildcatters or Zuckerberg-type nerds with no limits on or controls over what they were doing. To make this a little more concrete, it’s been treated as obvious from the beginning of the internet that certain kinds of cryptography count as munitions and the state is in charge of who you can give them to. I know from conversations I’ve had with people high up in the AI industry that a significant number of them say, we’re making decisions we shouldn’t be making. The public needs to be making these decisions or at least have a lot more visibility into them.
What I’m describing here is at a very high level of generality and leaves most of the key questions and decisions unsettled. But it’s at least the right framework.
But when I looked at the in-text citations and then at the footnotes to the study, I saw that they were to papers and studies authored, overwhelmingly, by the top-line weirdos I discussed in that post on Tuesday — Bostrom, MacAskill, Yudkowsky, Ord — as well as the cluster of institutes and centers which have grown up around their thinking and in their orbit. There’s a real garbage-in, garbage-out problem here. Or to put it slightly differently, the problem for me, as a reasonably intelligent person who is trying to understand and evaluate this as best I can with a view toward public accountability and the well-being of society, is a lack of reliable narrators.
I’ve seen too much about what these folks’ values are (highly questionable in ethical terms), and how they think (sloppy and even juvenile, a kind of scientistic wish- and doom-casting) to take their word for really anything. But I don’t know who the reliable narrators are. And that leaves my analysis, any guidance I can provide to myself or to others, very incomplete and inadequate.
To come at this slightly differently, places like Rand exist in the U.S. state-planning, national security space to look at the sciences, geo-politics, military technology, take the best info, chop it up and organize it so policy makers can make informed decisions. If you know the history of the last 80 years, the results of that process haven’t always been great. But that’s the idea. And reading this I experience a kind of needle-scratch moment where I suddenly realize, wait, you’re operating on the basis of what those dudes think and those are the baseline assumptions for decision-making? WTF?
What does seem clear to me is what I said in that earlier post and what a decent number of those issuing the warnings agree on: that we have a rapidly advancing technology that is already having a lot of deleterious effects on society and at a minimum could cause severe harms in the wrong hands and it’s being developed, controlled and deployed by a stereotypically start-up-y, move-fast-and-break-things kind of culture and mentality. And that alone is enough to tell us we’ve got a major problem and enough to act on without answering the big cosmic and totalizing questions.