A former researcher at the artificial intelligence company Anthropic has voiced stark warnings about the existential risks posed by AI, stating that humanity faces a greater than 10% chance of extinction within the next decade if current development trends continue.
Jacob Coxon, who recently resigned from his position, expressed fears that the rapid, unregulated pursuit of self-improving superintelligence is akin to "gambling with our lives." He emphasized that the problem is solvable but requires a significant slowdown and global cooperation.
"People are begging for regulation," Coxon said in an interview. "I think it’s just a matter of — international — somebody stepping in and being able to ensure that we don’t race ourselves into extinction. The problem is eminently solvable."
Coxon, who spent three years conducting pretraining research at both OpenAI and Anthropic, believes that the threat from current AI models is minimal. However, he highlighted the accelerating pace of recursive self-improvement, where AI systems enhance their own intelligence, as the primary source of concern. He noted that some in the industry believe this rapid self-improvement could lead to AI takeover scenarios within a year, or by the end of the decade at the latest.
Evan Hubinger, Anthropic's alignment science lead, corroborated Coxon's concerns, writing on X that "we really do earnestly believe AI could kill all humans!" Hubinger stated his personal belief that there is "more than 10% within the next decade" for AI to cause human extinction, adding that while Anthropic is trying its best, "we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to."
Coxon described AI as potentially the most dangerous technology humanity has ever created and likened the current situation to an international arms race. He argued that international oversight is crucial to prevent disastrous outcomes that could surpass any previous human conflict or activity.
Despite the dire warnings, Coxon maintained that a path to preventing an AI-driven catastrophe exists. He suggested that a concerted effort to slow down development and implement safeguards, rather than succumbing to an inevitable race, could avert disaster. He also noted that many technology leaders are seriously advocating for such a slowdown, and their concerns should not be dismissed as mere marketing.
Coxon acknowledged the immense potential benefits of AI, stating that humanity is "on the doorstep of tremendous discoveries." However, he stressed that the speed of development necessitates moderation and careful consideration to ensure these advancements do not lead to unintended and catastrophic consequences.