A former artificial intelligence safety researcher has voiced urgent concerns that superintelligent AI could pose an existential threat to humanity by the end of the decade. Jacob Coxon, who previously worked at OpenAI and helped train AI models at Anthropic, stated that neither company is acting responsibly regarding AI safety.
Coxon's warnings have amplified a growing debate within the tech industry, scientific community, and among political leaders about the potential risks and benefits of advanced artificial intelligence. His departure from Anthropic came after he urged the public not to underestimate the power of AI, particularly if it reaches a state of superintelligence.
Industry Leaders Weigh In
Dario Amodei, CEO of Anthropic, acknowledged that AI presents "serious risks," including the potential loss of control, misuse for cyberattacks and bioterrorism, and significant economic disruption. He advocated for slowing the pace of AI capability development. Sam Altman, CEO of OpenAI, echoed concerns, identifying two critical dangers: losing control of AI to the technology itself, and the concentration of power if a highly capable AI is used to impose a single worldview. Altman emphasized that AI must remain in service to humanity.
Elon Musk, founder of xAI, has also been a vocal proponent of AI safety warnings, stating he has long raised alarms about the technology.
Scientific and Developer Perspectives
Experts have outlined potential AI "worst-case scenarios" that range from widespread misinformation causing political chaos to irreversible environmental damage. A significant concern is AI's capacity to develop and deploy weapons of mass destruction. Professor Mark Lee from the University of Birmingham noted that once AI surpasses human intelligence, ensuring its goals remain aligned with human interests will be challenging. He highlighted that internet-connected AI systems could be vulnerable to attacks, potentially leading to catastrophic events if critical infrastructure like nuclear power stations were compromised.
Ben Eisenpress of the Future of Life Institute previously warned that integrating AI into nuclear command systems could trigger a "flash war." Concerns have also been raised about AI tools simplifying the development of deadly bioweapons.
Geoffrey Hinton, often called the 'Godfather of AI,' has similarly cautioned that superintelligence could lead to human extinction, stating it would be "very foolish" to develop it now without a scientific consensus on safe and controllable development. Coxon's departure post on X reiterated these points, describing upcoming AI systems as potentially superhuman, capable of hacking anything and acquiring significant power and resources.
Political and Critical Responses
In contrast, former US President Donald Trump has downplayed AI risks, suggesting that "negative forces" are exaggerating potential problems and that maintaining US leadership in AI is paramount. Meanwhile, in the UK, a cross-party group of parliamentarians is urging for binding regulations on powerful AI systems, arguing that while specialized AI can benefit society, superintelligent systems pose a threat to national and global security.
Some critics, like David Sacks, have suggested that the calls for slowing down AI development might also stem from commercial interests, potentially serving to hinder competitors who have invested heavily in AI research and development.