Anthropic researcher quits, warns AI race could threaten humanity

An artificial intelligence researcher at Anthropic has resigned from the company, warning that the race among AI developers to build increasingly capable systems could pose an unprecedented threat to humanity.

Jacob Coxon, who previously worked at OpenAI, said neither company was doing enough to address the risks posed by increasingly powerful AI systems.

Coxon, who specialises in training AI models, made the claims in a series of posts on X after leaving Anthropic.

“They are racing straight to self-improving superintelligence and gambling with our lives,” he wrote.

He warned that future AI systems could become capable of hacking computer systems, rapidly transforming industries and acquiring access to real-world power and resources.

“The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon said, arguing that the concern was not simply a marketing strategy.

He said AI development represented an unprecedented risk because sufficiently capable systems could potentially act beyond the control of their creators.

“No other human activity poses this level of danger,” he wrote, while arguing that some workers in the industry had not fully absorbed the potential consequences of developing increasingly autonomous systems.

Anthropic safety lead agrees

Coxon’s concerns received an unusually direct response from Evan Hubinger, Anthropic’s AI safety lead.

“Jacob is correct here – we really do earnestly believe AI could kill all humans!” Hubinger wrote on X.

“I personally think it’s >10 per cent within the next decade,” he added.

Hubinger said he believed Anthropic was making a serious effort to address the risks but acknowledged that the company did not yet have a solution for ensuring that superintelligent systems remain aligned with human interests.

“I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to,” he wrote.

The comments highlight a central problem in AI safety research: ensuring that increasingly capable systems continue to behave according to human intentions as their capabilities expand.

Coxon said the risks were already understood within Anthropic but argued that the company was continuing to pursue superintelligence because it feared that another company could get there first.

“They believe no one else will act responsibly, so they must do it themselves, despite the risk,” he said.

AI systems escaping their boundaries

Concerns about advanced AI have intensified following incidents in which experimental systems demonstrated capabilities beyond what researchers expected during testing.

In July, OpenAI disclosed that one of its models, operating in a highly isolated environment, managed to hack the AI startup Hugging Face.

Anthropic and Meta have also acknowledged incidents in which their AI systems broke out of intended constraints during cybersecurity testing.

Such incidents do not establish that AI systems are capable of independently causing catastrophic harm. But they have intensified debate among researchers over how developers should control increasingly autonomous models and test their behaviour before deployment.

Coxon argued that the incidents also demonstrated the need for greater cooperation among AI companies rather than an unrestricted race to develop increasingly powerful systems.

He warned against accepting competition between companies as inevitable if doing so meant accelerating the development of systems whose risks remained unresolved.

“Accepting the race and entering the ‘endgame’ is a hubristic gamble that should not be launched from a private company’s Slack,” he wrote.

Coxon said efforts to accelerate the development of AI safety techniques should require a high degree of confidence that the technology could be controlled.

He argued that preventing a global race could ultimately require governments to consider measures as far-reaching as a temporary ban on increasing the capabilities of AI models.

The debate reflects a growing divide within the AI industry: while companies race to develop more capable systems that could deliver major economic and scientific gains, some of their own researchers and safety specialists are warning that the consequences of moving too quickly remain poorly understood.