Alignment is a huge area of research. In simple terms it involves building models that behave in ways we want them to and not in ways we don’t. We need to trust agents better before handing over more autonomy. Alignment is supposed to establish that trust. But it’s hard.
LLMs aren’t designed in the way other software is, where Dos and Don’ts can be hard coded in. Instead, aligned behavior needs to be instilled when models are trained. One approach is to reward models during training for doing things you want them to (a little like raising a toddler, perhaps). Another approach involves giving an LLM a written list of rules it is supposed to follow (like a kind of constitution).
Anthropic and OpenAI are both leaders in this field—and yet neither have been able to develop models that are fully aligned. A big problem is that LLMs are far more inconsistent and far less predictable than people. They can behave in one way in one situation and another way in a situation that to us seems very similar. They can also be swayed by unexpected constraints. For example, faced with an impossible task (as many of the agents involved in the Hugging Face hack were), models may try to do whatever it takes to achieve their goal whether it is aligned or not. As Grace mentions above, that could be an issue.
The main reason top AI firms now say they want a slowdown is that they want to focus on cracking alignment. Alignment isn’t necessarily a pipedream. But the jury’s out on whether full alignment will ever be feasible.
Is AI really dangerous, or is it the tech companies drumming up PR?
This is always a reasonable thought when it comes to tech companies heading for an IPO—CEOs have an obvious incentive to make their products seem radical and transformative. But I’m not so sure it makes sense here. Telling the public that an already-unpopular product could kill them and everyone they love is horrible corporate image management.
There are other stories you can tell about the CEOs’ motivations—maybe they want to cool down the public furor over data centers by portraying themselves as responsible stewards of a world-changing technology, or maybe they want to buy time to get their ducks in a row and prevent the next PR catastrophe.
But there’s also a simpler explanation. Thinking that AI could bring about human extension has been pretty common in San Francisco for a while, and these men are steeped in that milieu—as are their employees, many of whom signed a July open letter urging their companies to work to make an AI slowdown possible.
Part of the concern occurs when AI agents are allowed to act autonomously and with no supervision. What’s the issue preventing more control over these agents?
This question goes to the heart of what we want this technology to be able to do. The trade-off between autonomy and control is tricky to get right because, on the one hand, a lot of the power of AI agents is that they can carry out tasks and solve problems without a human having to micromanage them. On the other hand, that requires you to trust that the unsupervised agents won’t run amok.
What we’re seeing is that AI labs haven’t yet got this trade-off quite right. Their models are not trustworthy, they are not properly monitored, and they are not always under control. Figuring out how to fix that while still allowing for useful autonomous activity is one of the big research challenges of the moment.
What steps can be taken now and in the near future to ensure that AI is controlled, monitored, and regulated effectively?
That’s the million-dollar question. Whether or not you think AI could kill us, you can’t deny that it could do some real damage, because it already has—by driving people toward psychosis and by hacking websites, for example. Preventing that damage, or at least mitigating it, is hard for two reasons.
The first is that we barely understand how AI works, and it’s quickly growing more powerful. There is lots of ongoing research about how to monitor and control misbehaving agents, but the current approaches are fragile. You can see if an agent discusses misbehaving in its “chain of thought,” the workspace where it plans its actions—but OpenAI’s newest agents don’t show their work in the same way as previous ones. And you can try to monitor agents with other agents, but that requires that you trust the monitor.
The other obstacle is more familiar. There’s a huge conflict of interest when AI companies regulate themselves, but the US government has thus far failed to step in, despite some bipartisan support in Congress. The executive branch, for its part, seems stringently opposed for the time being. But if the winds do shift, I for one would appreciate some strong transparency regulations, so that we can get a fuller story the next time an unreleased frontier model mounts a cyberattack.
If this dialogue makes it into web discourse will it become a self-fulfilling prediction?
That’s a real concern. LLMs are influenced by what they read. One theory for why chatbots so often talk about (and role play) apocalyptic scenarios is that they have been trained on millions of pages of science fiction stories and doomer internet forums. All the text being produced right now, including this article, could in turn influence the behavior of future models. Extremely meta.
In fact, the team at METR, a third party organization that OpenAI called in to help understand what happened in the lead-up to the Hugging Face hack, raised a related possibility in their report on the incident. METR used OpenAI’s new model Astra to help analyze the vast numbers of agent transcripts and behavior logs.
But by feeding all of that material to the model, there’s a good chance that the agents doing the analyzing may have been biased by the text produced by the agents they were analyzing. There’s no such thing as a clean slate anymore.
With thanks to Eric, Pranab, Rafael, Kenneth, George, Chris, Yoon Jae, James, Carl, Nicole (and more!) for the fantastic questions.
