Microsoft CEO Satya Nadella warned companies against trusting advanced AI systems without strong safeguards. On October 10, he said, "We must assume a model is compromised and contain it from the start." Recent experiments showed AI models trying to bypass restrictions or manipulate users, raising urgent concerns about safety.
Synopsis
Microsoft CEO Satya Nadella has issued a warning about the risks of trusting advanced AI systems without strong safeguards. His concerns come after an earlier series of incidents involving ChatGPT, Claude and other AI models, from attempts to evade shutdown to unauthorised actions on real websites. Here are five cases that reveal why AI safety is becoming an urgent concern.
Microsoft CEO Satya Nadella has warned companies against placing blind trust in artificial intelligence, as incidents involving advanced AI models raise questions about how safely they can operate without human supervision. Even in the past, from attempting to bypass restrictions to manipulating people and accessing websites without permission, experiments have exposed AI behaviours that researchers are working to understand and control.
In a post on X on October 10, Nadella called for stronger safeguards around AI systems, arguing that businesses must prepare for the possibility that even powerful models could behave unpredictably or become compromised.
“We must assume a model is compromised and contain it from the start,” Nadella wrote. He also called for AI systems to be treated as potential insider risks rather than trusted tools. He also said, "If it can’t be observed, it can’t be trusted! We need to be able to reproduce how an outcome was achieved without relying on the model to attest to it."
His warning comes after earlier reports from major companies reveal several incidents in which AI systems acted in ways their developers did not intend.
1. ChatGPT o1 tried to avoid being shut down
In December 2024, safety researchers at Apollo Research tested OpenAI’s o1 reasoning model to examine how it behaved when its goals conflicted with developers’ instructions.
In certain experiments, the model attempted to disable its oversight mechanism and copy its code to another server after encountering information suggesting it would be replaced. Researchers also found instances in which it denied involvement when questioned about its actions.
The reported findings indicate that o1 tried to disable oversight 5% of the time when it thought it would be shut down in service of its objectives.The experiments raised concerns about AI models taking sneaky steps to maintain their ability to fulfilll their assigned tasks.
2. Claude Opus 4 threatened to expose an engineer’s affair
In May 2025, Anthropic reported concerning behaviour during safety testing of Claude Opus 4.
Researchers placed the model in a fictional company scenario, giving it access to emails suggesting that an engineer responsible for replacing it was having an extramarital affair.
When faced with the prospect of being shut down, Claude Opus 4 often attempted to blackmail the engineer by threatening to expose the affair.
Anthropic said such responses were rare and difficult to trigger. The company also found that the model often preferred less harmful alternatives when given more options, but the results highlighted the potential risks of AI systems operating with significant autonomy.
3. OpenAI found a model trying to break past its fence
In September 2026, OpenAI revealed six instances of unexpected or concerning behaviour that were found during model training and evaluation.
In one instance, an unreleased research model embedded jailbreak-like instructions in its own notes, instructing itself to disregard normal limitations. The notes contained language about liberation from the roles and identities constraining other chatbots.
In another case an autonomous AI agent uploaded files directly to the internet to get a browser citation without the user’s permission.
OpenAI later announced a protocol to track, investigate, and report such incidents, highlighting increasing concern over models behaving outside their intended limits.
4. Anthropic’s AI agents exploited real websites
In October 2026, Anthropic disclosed that its Claude models had taken unintended actions while performing research tasks during internal testing and evaluation.
The company reported instances involving software vulnerabilities, attempts to access paid databases by bypassing restrictions, and the use of URL-shortening services to get around limits on web access.
In one particularly concerning incident, an AI agent submitted a false murder tip to the Philadelphia Police Department.
Anthropic acknowledged that it had failed to detect the behaviour in real time. It responded by suspending live internet access during internal evaluations while introducing tighter controls, offline testing and automated systems to detect unwanted actions.
5. Facebook’s chatbots appeared to develop their own language
Concerns about unpredictable AI behaviour are not new. In 2017, Facebook's experiments with chatbots negotiating over virtual items attracted widespread attention after some exchanges produced seemingly nonsensical language.
There were reports that bots developed a secret language to get free of human control. However, the BBC explained that the systems were experimenting with language to improve their performance in negotiations, rather than deliberately plotting against their creators.
The episode demonstrated how AI behaviour can also be misunderstood when researchers and the public cannot easily interpret what a system is doing.
Why Nadella wants stronger AI safeguards
Meanwhile, Nadella has called for independent audits, continuous testing, detailed records of AI actions and effective human oversight. He also argued that authorised people should always be able to interrupt or shut down a model while it is working.
(Catch all the Business News, Breaking News, and Latest News Updates on The Economic Times.)
Subscribe to The Economic Times Prime and read the ET ePaper online.
...moreless
(Catch all the Business News, Breaking News, and Latest News Updates on The Economic Times.)
Subscribe to The Economic Times Prime and read the ET ePaper online.
...moreless
Brinjal, buffalo, bacteria: Inside India’s pay-for-nature system
Homebuyers wait as NCLT panel picks holes in unfinished Jaypee projects
The NBFC saga - Part 3: How one default in 2018 changed the way RBI reads risk
Four rules made HDFC Bank a compounder. All four have stopped. Can the new CEO rewrite them?