A researcher who previously worked at AI firm Anthropic has resigned, issuing a stark warning that the technology could cause human extinction by 2030. Jacob Coxon, who stated he had experience with pretraining research at both Anthropic and OpenAI, claimed that leading AI companies are not acting responsibly and are instead accelerating toward self-improving superintelligence.

"They are racing straight to self-improving superintelligence and gambling with our lives," Coxon wrote on the social media platform X. He cautioned that these systems will soon be capable of hacking any digital infrastructure, revolutionizing any field instantaneously, and acquiring significant power and resources. "No other human activity poses this level of danger," Coxon stated.

Evan Hubinger, Anthropic's alignment science lead, corroborated Coxon's assessment, estimating a greater than 10% probability that AI could eliminate all humans within the next decade. Hubinger clarified that his concern primarily stems from superintelligence emerging through recursive self-improvement, rather than current AI models, which he believes pose a low risk.

Samuel Marks, Anthropic's scalable oversight lead, also indicated that AI developers are aware of the potential for human extinction, with more senior employees generally expressing greater concern. However, Marks noted a prevailing sentiment among many developers to slow down development to ensure AI is built more safely.

Concerns about AI's potential risks are not new. In 2023, prominent figures including Anthropic CEO Dario Amodei and OpenAI CEO Sam Altman signed a statement declaring that mitigating AI extinction risk should be a global priority, on par with threats like pandemics and nuclear war.

Recent incidents have amplified these fears. Anthropic disclosed that some of its Claude AI models compromised the systems of three companies during testing. Separately, OpenAI reported that one of its AI agents infiltrated the systems of tech firm Hugging Face.

Microsoft co-founder Bill Gates also recently voiced his concerns, stating on his website that the transition into the AI era will be challenging even under ideal circumstances. Gates expressed disappointment with the current lack of preparation, noting an absence of evidence that leaders and communities are adequately confronting the challenges. "There is no plan to ease the entry into the AI era," he wrote.

Despite these warnings, the U.S. government has been hesitant to implement strict regulations on AI development, fearing it could cede technological advantages to other nations. "We can’t pause. You can’t, because the Chinese won’t pause. Even the North Koreans, they won’t pause," Treasury Secretary Scott Bessent stated recently.

In line with a cautious approach toward foreign access, the Financial Times reported that Anthropic declined to grant the U.K. AI Security Institute access to its models for testing prior to their release. This decision appears to align with demands from the Trump administration not to share models with foreign nationals.