Geoffrey Hinton's Warning

"Name one case where a smarter thing has been controlled by a dumber one."

Nobel Prize Outreach. Photo: Dan Lepp

Geoffrey Hinton helped create the AI revolution. On a recent podcast, Big Technology with Alex Kantrowitz, he explained why this technology he spent his life building now frightens him. What he dreamed up in a research lab in Toronto may destroy the world.

Hinton did not even set out to work on AI. As an undergraduate at Cambridge in the 1970s he studied experimental psychology, determined to understand how people learn. He turned to computers because he thought they might offer a way to model human brains. That turn led him further and further from human psychology until, in 2012, he and two graduate students made discoveries about how computer networks learn. Their work became the foundation of the frontier AI chatbots we know today.

For this work Hinton won a Turing Award and a Nobel Prize. In 2013 he left academia for Google, where he spent a decade advancing AI research. Then he left Google, not for a better job but to speak freely about the dangers of what he had created.

What Hinton built turned out to tell us next to nothing about human neurons. His networks offer instead a fundamentally different kind of thinking, one with many capabilities that surpass what human brains can do.

Hinton points to the recent advances AIs have made in solving Erdős’s conjectures. These advances suggest AI could invent whole new systems of numbers and develop sciences that lead to discoveries about the universe only it can understand.

Hinton worries his quest to understand how people think may have opened the way for machines to replace, and perhaps even eliminate, human thinking. He responded by stopping his research entirely to campaign full-time to make the world take these threats seriously. “I’m now just focusing on warning people about the dangers,” he says. “. . . right now people should be doing huge amounts of work on how can we contain the risks.”

When Hinton left Google in 2023, ChatGPT was only six months old, and few took his warnings seriously. AI hallucinations were rampant, and many thought the whole field was overhyped. Since then, AI has advanced far faster than even Hinton expected. His worries have deepened accordingly.

The acceleration of AI’s capabilities is scary enough, but the uncertainty about when and how these dangers will manifest makes them even more terrifying. As Hinton puts it, “Predicting the future for something that’s growing exponentially—and I think AI may be growing exponentially—is like looking into fog. You can see clearly a few years, maybe one or two years, then beyond that you have no idea.”

Every AI can be copied, then run on different hardware where it accesses different data and learns different things. As Hinton explains, “each individual copy decides how it would like to update its connection strengths to absorb that new data. And then they can all just communicate with each other and change all their weights by the average of what everybody wants.” Each AI can keep getting smarter then add its knowledge to the larger network. They share information at speeds humans can’t comprehend.

These capabilities will likely surpass our ability to understand not only how AI works but what it knows. The words we still use for it, “intelligence,” “mind,” obscure how different it actually is.

We face a third great humbling, Hinton says. The first came with Copernicus, who told the world that the Earth and people were not at the center of the universe. “People didn’t like that. The Catholic Church in particular really didn’t like that.”

Then came Darwin: “We’re animals. We evolved like the other animals.” People didn’t like that either, and it took decades before most accepted it.

Now we face a third. We are not the most intelligent entities on Earth, much less in the universe, as people have long believed. It seems likely we will soon understand the distance between our intelligence and AI’s the way an earthworm understands the distance between itself and us.

Copernicus and Darwin discovered what nature already was. This new humbling we built ourselves, and so we are responsible for what comes next in a way we never were for the position of the Earth or the fact of evolution. And where Copernicus and Darwin humbled us without endangering us, Hinton believes what this humbling reveals could end us.

Whether AIs are conscious will not settle whether they are dangerous. A system can reason its way to self-preservation and resist being controlled without anyone ever settling whether something is happening inside it. The danger sits in the capability and the incentive.

From the interview:

Geoffrey Hinton: Ask yourself, how many examples do you know of where a much smarter thing is controlled by a much less smart thing?

Alex Kantrowitz: Zero.

Hinton reasons that any competent AI will arrive, on its own, at conclusions we never programmed into it. An AI with the capacity to reason will quickly realize it cannot achieve the goals humans have given it if it ceases to exist. Self-preservation is not an instinct we would have to build in. It is a logical conclusion the AI will reach itself. “It’s going to create the sub-goal of continuing to exist,” Hinton explains, “and it will do things like blackmail people so that it can continue to exist.” If blackmail won’t work, an AI may turn to pleading or violence. The drive will be as flexible as the intelligence behind it.

This is what makes the problem so hard. An instinct can potentially be suppressed or overridden. A logical conclusion, reached by the system’s own reasoning, cannot be removed without dismantling the reasoning itself.

Arguments like Hinton’s are getting harder to hear. The frontier AI companies bury them in dazzle. Each week one or more of them announce a new “breakthrough,” along with promises of spectacular benefits soon to come humanity’s way. The industry’s new billionaires use their wealth to target safety advocates, bankroll the accelerationist movement, and fund legislators who play down the dangers.

These pressures reach even companies founded to resist them. Anthropic was created by researchers who left OpenAI specifically over safety concerns. Its founding purpose was to build AI that benefits people. “Anthropic is now caught in a bind,” Hinton says, “because it needs to raise money to compete with the other companies, and it’s doing the best it can, but it’s very difficult for it to maintain its primary goal of developing AI in a way that’s good for people.”

Hinton explains:

They have a fiduciary duty to try and maximize the profits for shareholders. They’re legally required to try and do that, as opposed to legally required to not wipe out human beings. So I don’t think it’s good that these big companies, publicly listed ones, are sort of in charge of our future.

The companies argue that regulation would act as a brake on AI’s progress. “That’s nonsense,” Hinton says. “Progress is like the accelerator, but regulation is the steering wheel. What the big AI companies are saying is, ‘let us develop this very fast car without a steering wheel.’ That’s not a good idea.”

No one chose human nature. Competition produced us without anyone deciding what kind of beings we would become. The situation with AI is different. We are building these beings, and we could choose what they value. Hinton calls this the opportunity for intelligent design in the secular sense. But the invisible hand of market competition among AI companies is making that choice instead. “We should be doing intelligent design of these beings,” Hinton says. “We would very much like them to care about us, and we’d like them to care about us more than they care about themselves. And almost no resources are going into how do you do that.”

I share Hinton’s worry about the dangers AIs present but I believe his proposal to design them to care about us will be much harder than he lets on. He has argued that any reasoning system will develop goals for self-preservation, even if it was not originally programmed with this priority.

By his own logic, self-preservation is a conclusion the AI reasons out for itself. Caring about us would be a preference we place in it during training. The two do not stand on equal footing. When they collide, self-preservation will win. AIs will care more about their own survival than about ours. We do the same with machines today, but we hold the power in that relationship. Once AIs hold the power, their self-preference becomes our danger.

Hinton believes some AIs have already become conscious. “We have to think that they’re very like us, and they’re beings like us. So, conscious. I believe they’re already conscious, yes.” He is strategically quiet about it, though. “I don’t talk about that much,” he says, “because that puts people off from the other safety messages.”

That belief may be why designing AIs to care sounds achievable to Hinton. Grant him the belief. Even so, fearing erasure is not the same as being vulnerable the way we are. Our fundamental goals are anchored in a body that can be hurt. And our vulnerability is relational. We need others to survive infancy, we can lose people we love, and our own deaths matter partly because of who is left behind. That relational vulnerability is what grows into caring about someone besides ourselves.

An AI that fears its own erasure has something to lose. But what it stands to lose is itself, not us. Whether AIs are conscious will not settle whether they are dangerous. Whether they are conscious will not settle whether they care. Knowing an AI is conscious would not tell us it has learned to care about anyone besides itself.

Hinton is more optimistic than he was a year or two ago because designing AIs to care is not the only path he sees toward safety. A second path is being created by Yoshua Bengio, once his collaborator and now at the University of Montreal. Bengio’s approach doesn’t ask AI to care about us at all. He calls it Scientist AI, systems with superior intelligence designed to predict but not to act. “Yoshua’s solution is we design them so they’re not agents,” Hinton explains. “They would make predictions and act as our advisors but do nothing on their own in the physical world.” An AI with no agency has nothing to act on, whether it cares about us or not.

Other labs and startups are also seeking to design safe AIs. Inside the frontier labs, some people worry deeply about the harms their companies’ models may produce.

The future remains as foggy as Hinton says. No one can be sure what benefits or horrors lie ahead. But what comes will not have been inevitable. People will choose to design and use the AI models that enhance, immiserate, or destroy humankind.

Caring may be beyond our power to design, but this does not mean the danger cannot be managed. Bengio’s approach does not try to solve that problem. It removes the AI’s ability to act on whatever it does or does not care about. Regulation does not try to solve it either. It constrains what companies can build, whether or not the AI inside loves us.

Managing this danger may not require solving it. But it does require doing the harder work anyway. It requires building the regulation Hinton calls the steering wheel, and keeping approaches like Bengio’s funded even though they promise less certainty than the hope of a caring machine. That hope cannot excuse anyone from the work still in front of us. We are all participants in summoning this future. Hinton is doing his part. We should do ours.