Jacob Coxon, a 27-year-old researcher who worked with OpenAI and Anthropic, warned that AI automation could end humanity. He told the New York City Council that AI systems might soon conduct their own research. "It is more likely than not that humanity loses control to these AIs," Coxon said.
A 27-year-old AI researcher who has worked with both OpenAI and Anthropic has issued a stark warning about where the race to build more powerful artificial intelligence could lead. Jacob Coxon says AI systems capable of conducting their own research could arrive within the next few years, leaving humans with far less control over the technology.
Coxon, who recently resigned from Anthropic after working as a capabilities researcher, made the comments while appearing before the New York City Council during a hearing on the risks posed by AI. According to Coxon, automating AI research has become a major objective for leading AI companies. He said his experience at OpenAI gave him an inside view of how quickly the technology was progressing.
"When I was at OpenAI, we had milestones for an automated AI researcher, around 2027, 2028, and we are on track or beating these expectations," Coxon said.
He said this timeline could put AI much closer to conducting its own research than many people realise. If AI systems begin developing better AI systems themselves, humans could gradually have less involvement in decisions about how the technology advances.
‘I was working to automate myself’
Coxon described the development in personal terms, saying the work he was doing could eventually replace part of his own role as a researcher. "At my last job, I was, in some sense, working to automate myself," he said. He also raised concerns about how much AI is already being used to write computer code. Coxon told lawmakers that a large share of code written today is generated by AI and claimed people are not always checking it as carefully as they previously did. His bigger concern is what happens when AI becomes capable of doing much more than assisting humans.
"Humans will be much further out of the loop than we are today, and today is already not great," Coxon said.
Coxon said his experience at Anthropic and OpenAI had left him concerned about the safeguards surrounding increasingly powerful AI systems. He told the hearing that he believes the consequences could go as far as the loss of human control.
"It is more likely than not that humanity loses control to these AIs, and it could end in human extinction," he said. He also criticised the pace at which AI companies are developing their technology, arguing that the approach used by startups may not be appropriate for systems that could have consequences on a global scale.
"The companies run on a startup mindset: move fast, break things, fix them later. That works for a photosharing app. It does not work for building the most powerful technology ever built," Coxon said.
More techies warn about AI risks
Coxon's comments come amid a growing number of warnings from people working in the AI industry. Anthropic safety researcher Evan Hubinger recently said he believes there is a greater than 10 percent chance AI could kill all humans within the next decade. He has also said the industry does not yet have a solution for keeping future superintelligent AI systems aligned with human interests.
Anthropic has itself assessed the possibility of highly capable AI carrying out automated research and development. In a recent safety report, the company said the current risk of such systems causing catastrophic harm was low, but acknowledged it was less confident in that assessment than before.
AI company leaders have also called for caution. Anthropic CEO Dario Amodei has urged the industry to slow the pace at which AI capabilities improve. OpenAI CEO Sam Altman has said he agrees that the development of frontier AI needs to be paced.
Elon Musk has made a similarly serious prediction, saying humans could lose control to AI within the next decade. He has also put the chance of AI wiping out humanity at between 10 per cent and 20 percent.
Coxon is now calling for greater transparency from AI companies and a slower approach to the development of increasingly autonomous systems. His warning comes as the industry moves towards AI that can do more of the research and development work that humans currently perform.