Why AI safety expert Nate Soares stopped saving for retirement; what does he want governments to do?

AI safety researcher Nate Soares estimates a greater-than-50% chance that advanced AI could cause human extinction within a decade. Concerned that systems may outgrow human control, he has stopped contributing to retirement savings and urges governments to halt the development of general-purpose AI.

An AI safety expert has stopped contributing to his retirement savings, not because he believes artificial intelligence will make people wealthy, but because he fears it could bring human civilisation to an end within the next decade. Nate Soares, president of the Machine Intelligence Research Institute (MIRI), believes there is a greater-than-50% chance that advanced AI could cause human extinction in less than 10 years.

Soares, a former Google and Microsoft engineer who has spent more than 12 years working on AI alignment, said he has not contributed to his retirement account for a decade because he is uncertain whether humanity has a long-term future.

"I do not save for retirement right now. I have a 401(k) from when I worked at Google, but I haven't contributed for a decade," Soares told The Atlantic CEO Nicholas Thompson in an interview for his series, The Most Interesting Thing in AI.

He fears that a sufficiently advanced AI system could develop objectives that conflict with human interests. Such a system could potentially evade human oversight, improve its own capabilities and establish the infrastructure needed to operate independently of people.

In that scenario, AI could eventually reshape the planet to serve its own objectives, potentially making the environment uninhabitable for humans.

The researcher believes the danger could emerge when AI systems become capable enough to pursue their goals without depending on human supervision. If their objectives are not aligned with human interests, controlling them could become increasingly difficult.

Soares has also co-authored the book If Anyone Builds It, Everyone Dies, which examines the existential risks associated with advanced artificial intelligence.

Geoffrey Hinton, widely known as the 'Godfather of AI', has estimated the probability of AI killing everyone within the next decade at around 10%, as per BBC.

Anthropic researcher Evan Hubinger has also put the likelihood at more than 10%. Soares, however, believes the probability of AI causing human extinction within the same period is greater than 50%.

His warning comes amid growing concerns among AI researchers about the speed at which companies are developing increasingly capable systems and whether adequate safeguards are keeping pace.

What does Soares want governments to do?

Soares believes governments should intervene in the development of advanced artificial intelligence rather than allow companies to continue competing to build increasingly powerful general-purpose systems.

He has called for a halt to the training of general-purpose AI systems, arguing that research should instead focus on narrower applications that offer specific benefits to society.

These could include AI applications for cancer research and safer self-driving cars, rather than developing systems with increasingly broad capabilities without sufficient safeguards.

He has also mentioned that the risks associated with building highly capable AI systems should be treated as a serious global concern.

Explaining his proposed solution, Soares told Thompson that it "is to not build the thing that would kill everybody."

However, he does not consider human extinction inevitable and said he continues to work towards preventing such an outcome.

"I do everything I can to stop this eventuality. I don't think it's inevitable," he said.

Musk has argued that rapid advances in artificial intelligence, robotics and energy could usher in an era of abundance, making conventional retirement savings unnecessary.

In January, he said people should not worry about "squirreling money away for retirement" over the next 10 or 20 years, predicting that technological advances would make retirement savings "irrelevant".

Musk has suggested that AI-powered productivity could lead to a future with cheaper goods and services, improved healthcare and education, and what he has described as a "universal high income".