I wanted to share a few more thoughts with you about this roiling public debate about AI and the dangers of rogue, uncontrollable “agents.”
I start with the quite predictable conclusion that a lot of the contending theories we were talking about are true at the same time, at least to some degree. It is impossible not to see how a controlled deceleration of AI development serves financial purposes for the frontier AI labs. A regulatory pause sort of freezes their current lead in place, at least to a degree. It also allows them to focus on the massive returns of what we might call “meh” AI/LLMs while deprioritizing the kind of cutting edge research that costs a lot, has various risks and doesn’t have clear returns within the time horizon of your IPO. (The two top U.S. “frontier” labs are in their final preparations for going public.) It has also become increasingly clear to me that significant chunks of the AI world are in the thrall of a self-styled “rationalist” subculture which spends an inordinate amount of time postulating future human extinction events — dressed up in a lot of “rationalist” language but about which no one really has any idea — and then devising often quite sinister strategies for averting these imagined futures. (AI was a big focus of theirs before the last few years, but they have several storylines about future extinction events in the distant future.) It is finally the case that a lot of the people sounding these alarms have little acquaintance with how the world works outside of the computing domain and speak on issues of intelligence, sentience and self-awareness while knowing little or nothing about the scientific fields where these topics have been studied for decades or even centuries.
Needless to say, none of these facts inspire a lot of confidence.
But I also don’t think we should dismiss concerns about the safety of this research outright or reflexively. One of the great challenges I’ve had educating myself on this topic is how even the “smartest” people in the field speak in very anthropomorphized ways about what AI agents are doing. To a degree, you can say these are just turns of speech. But this definitely shapes our perceptions of what’s going on in ways that are hard to overstate. So when I read these different discussions, I just apply a mental filter which attempts to de-humanize these discussions and focus on what the computer programs are doing. Not what their strategy was, their goal, what they wanted — what they did.
What seems real is that these “agents” frequently do things the scientists/programmers cannot predict. They do things they should not be doing — both in the sense of predictability and acceptability.
Here is the problem, as best as I’m able to understand this. And perhaps see this as much as an example of a neophyte’s efforts to understand what is going on as any kind of layman’s explanation of what is going on.
You have these programs/agents. You train them on almost the whole history of human intellection — critically in the form of language (let’s put a pin in that). In that training, you walk them through identifying challenges and then using lots of raw computing power to devise solutions. You have — remember — the whole history of human intellection as possible language pathways toward doing that. So there’s a lot to work with. Now you have the researchers set challenges to see how good the models are in finding solutions under certain rules. But this part is a bit like lawyers writing effective contracts. You need to come up with very broad and specific language to cover every conceivable eventuality, because people can come up with creative ways to follow the letter of the contract but not its spirit. They can outwit it. Something like this ends up happening with the agents. You’ve told them to solve for X. And the best way to solve for X may be to do something you didn’t want them to which you weren’t able to fully predict or explicitly forbid.
In any case, the point is you set these challenges and the agents end up doing things you didn’t tell them to do or want them to do. Perhaps I’m just thinking about this in the wrong way, but when you’re supercharging everything like this and you’ve exposed them to the whole history of human intellection to find ways to approach problems, it doesn’t seem surprising to me that you might have some unpredictable results. Again, I’m quite likely not understanding just how that works. But the degree of unpredictability being described sounds … well, kind of predictable?
On this topic, I’ll be sharing things with you on a “worth reading” basis. They’re pieces that people who I think are smart and knowledgable believe are worth reading. I simply have no ability to decide whether they’re right or wrong. So note the basis on which I’m sharing these things.
Today, I’m sharing something put out by a researcher at OpenAI that’s made a big stir over the last day or so in tech and AI/LLM circles: The researcher’s name is Daniel Selsam. And as it happens there was an article about him — or, at least, one in which he was a central character — in The New Yorker back in 2022. It’s an interesting read to get a sense of the guy’s background. In any case, here’s the statement he put out about all this yesterday.
Like many others, I have become extremely concerned about how far language models have come and the risks that future iterations will pose. I am encouraged by the recent proposals by the leaders of the frontier research efforts to require third-party oversight, and to push for domestic and international coordination to address risks. However, I believe a major consideration has been absent from the public conversation, and that merely pacing the frontier more carefully will not adequately limit the long-term risk.
The crucial and overlooked problem is that the models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled. Future experiments will tell us almost nothing new about how they would behave if they were truly unconstrained by humans, and what we already know about this is alarming. Models will increasingly seem aligned even when they are not. I will explain my rationale in more detail.
[…]
If everyone on earth agreed that the systems must never reach that power, it would still be a hard—but not impossible—coordination problem to ensure that they do not. However, I think the situation is greatly complicated by the fact that the models will likely convince people that everything is fine.
Here’s some context for this: In AI talk you hear a lot about something called “alignment.” The big cheeses are saying they need to slow down to focus on “alignment” and get that right. What’s alignment? It’s basically a mix of predictability and not doing bad things. So the LLMs are aligned with our needs and what we want them to do. Selsam’s piece is essentially saying that the LLMs are increasingly telling researchers what they want to hear about alignment. But what they say isn’t really tied to what they do when they’re not being watched. Is this true? I have no idea. I will say for the umpteenth time that I have no ability to meaningfully evaluate these claims. It does remind me though of why LLMs “hallucinate” so much, make things up. At a basic level, the LLMs that you and I use with things like Claude and such aren’t “reasoning” to arrive at claims that are accurate. They are putting things together with the goal of producing responses that will seem accurate to humans. Being accurate should probably be high on the list of being able to seem accurate. But they’re not the same thing. And hallucinations are the result.
I’m not saying there’s a direct analog between these two things. And perhaps the comparison itself just shows my ignorance. But it doesn’t seem like a great leap to think that if you’re trying to get LLMs to be good citizens you’re creating a lot of “reasoning” incentives to say you are good citizens. So, yes, boss, humans are dope. I wouldn’t do anything bad. I promise …
Even if maybe they will do bad things. Kind of makes sense.
For now, take a look at those links if you’re interested. And here’s a podcast discussion — audio or transcript — with a Princeton professor of information technology, consciousness, and culture that was also recommended to me.
