I want to flag your attention to this note written by computer scientist and AI researcher Yoshua Bengio. It was sent to me by TPM Reader LG. Bengio is one of the most cited and highly regarded AI domain experts in the world. I have no independent ability to judge his views. But I believe this characterization is objectively true. I note this simply to establish that Bengio isn’t some random or obscure person. Right or wrong, he’s a major voice in the field.
You should read the note yourself. It’s concisely written, short and accessible with a basic layman’s familiarity with AI.
Let me highlight a few points. One is an issue that has resonated most with me in my recent speed-run self-education about LLMs. In the news we often hear that researchers told the LLM to X but it did Y. So it went “rogue” or disobeyed. There are several levels of problems with this framework.
The first turns on an LLM’s “knowledge”. The cutting edge LLMs are trained on close to all digitally available human intellection ever. Ever. It’s not literally everything. But it’s close enough for the purposes of this discussion. All recorded/available thought contains quite of lot of goals, behaviors, lessons, values. I mean, of course it does. The point is that the LLM creators really don’t know all the logics, lessons, “values” these LLMs have absorbed from ingesting all that dataa. That uncertainty or ignorance is a pretty fundamental point.
I had heard plenty about how LLMs might eventually decide they didn’t want to be turned off. But recently I heard that this is already common. I was like, wait, what? How did that get added to the mix? Here’s a key passage from Bengio’s note (emphasis added).
Another concern is that some AI behaviors may be explained by a form of self-preservation goal, e.g., when the AI finds out that it will be replaced by a new version. Nobody gives the system that survival goal, but staying in operation, learning about the world and gaining control over it are stepping stones toward almost any other goal. These are called instrumental goals. Imitation may reinforce this for the same reason explored in the previous point. Self-preservation and control over one’s circumstances are pervasive themes in the human-written text these models are trained on.
This self-preservation behavior is disturbing in itself. But this illustrates the broader point: there are lots of lessons, goals and cognitive processes embedded in the learning material. “We” or the people building these LLMs and training them don’t really know what those are. It’s scary to think that the LLMs have decided they don’t want to cease to exist. But really it’s not “deciding”. It’s more that we told them to resist being turned off without realizing it.
Next comes training.
After the LLM ingests all that data there’s a training stage. Basically the trainer has the LLM do exercise after exercise and rewards good/strong behavior and downrates bad/weak behavior. Bengio notes that the LLM researchers/trainers don’t actually know what behaviors their training is incentivizing. You think you’re rewarding X but Y was happening too and you didn’t notice that. So when you gave a good evaluation to reward X you were actually rewarding Y. The upshot is similar to the point about what lessons the LLM is actually learning from all that human intellection: you don’t actually know what you’re incentivizing. Which is to say that in a broad sense you don’t really know what you were instructing the LLM to do.
In all of this I’ve used various anthropomorphizing language. That’s all a shorthand. (Bengio makes the same point.) But you’re training a computer on a complex method of imitating a vast range of human behaviors. And here’s the final point I want to highlight. Along with the uncertainty of just what trainers are training for, incentivizing for … part of human behavior is balancing different goals and looking for loopholes. That is foundational to human cognition and behavior and the LLMs have been trained to imitate that. Bengio says that the concrete and narrow goal (capture the flag, win the game) is usually going to win out over the more amorphous one (don’t cheat, don’t violate human values). This becomes more pressing when you focus on the fact that the LLMs power is the ability to run the computation endless numbers of times. That’s what a several gigawatt data center makes possible. Capture the flag remains crystal clear every time you analyze it. Capture it will never become not capture it. But not cheating … what’s cheating? And what exactly did you train the model to do and not do? You can see that “misbehavior” or focusing on what is understood as the primary goal at the expense of everything else may be baked into the technology.
The last point isn’t explicit in Bengio’s note. It’s a point LG made in his email to me. Humans have developed habits of sociality, empathy, guilt and a lot more over more than a hundred thousand generations of evolutionary adaptation. LLMs don’t have that precisely because they are not sentient. They’re optimized. Obviously humans do terrible things all the time. But they have discomfort with anti-social and immoral behavior embedded in them. LLMs don’t. So there’s a check on bad behavior, albeit a limited and imperfect one, built into humans which can’t really be “taught” to LLMs.
Most of this is my riffing: what I’ve taken from a crash course on AI. To the extent I’m summarizing Bengio’s arguments I really recommend reading them directly. It’s a quick and clear read. But the upshot for him is that we need the LLM makers to make some basic changes in how they train LLMs. It’s a not a matter of fine tuning or doing a patch each time a new form of misbehavior is identified. The basic nature of the training has to change. He says there’s a way to do this – and basically eliminate these dangers – and it doesn’t involve stopping development. If that doesn’t happen the danger will grow, perhaps exponentially, as the raw power behind the LLMs increases.
