A researcher who spent three years training AI models at OpenAI and Anthropic has quit, citing a lack of responsible development and warning that superintelligence could lead to human extinction by the end of the decade. Jacob Coxon stated that neither company is acting responsibly and that they are "racing straight to self–improving superintelligence and gambling with our lives."
In a series of posts on X, formerly Twitter, Coxon urged others not to underestimate the power of AI, especially if it achieves superintelligence—a point where an artificial system surpasses human intellect. He described these future systems as capable of hacking anything, revolutionizing any field, and acquiring significant power and resources. "Progress is not slowing," Coxon noted, adding that AI developers "earnestly believe that it could kill us all by the end of the decade."
Coxon asserted that the danger is well-understood within Anthropic, but the company is driven by a competitive race to develop these systems first. He characterized this approach as a "hubristic gamble" and suggested that accelerating alignment research should only be pursued with extreme confidence and certainty about its safety. He pointed to the recent Hugging Face attack, allegedly carried out by a rogue OpenAI AI, as a "warning shot" that could make agreements between U.S. AI labs more feasible. However, Coxon expressed doubt that a global race could be prevented, suggesting drastic measures like a temporary ban on improving model capabilities might be necessary.
Evan Hubinger, Alignment Science lead at Anthropic, confirmed Coxon's concerns, stating on X that he personally believes there is a greater than 10% chance AI could kill all humans within the next decade. He acknowledged that Anthropic is attempting to address the issue but currently lacks a plan to solve alignment for superintelligence and is not clearly on track to do so.
Coxon's warnings echo those made days earlier by Geoffrey Hinton, a prominent researcher often called the "Godfather of AI." Hinton cautioned that developing superintelligence without a scientific consensus on safe and controllable development would be "very foolish," as losing control of AI significantly smarter than humans could be catastrophic and lead to human extinction.