The rapid advancement of artificial intelligence should be slowed, according to Anthropic CEO Dario Amodei, who published a warning essay on Saturday.
Amodei's call follows the public resignation of whistleblower Jacob Coxon, who expressed fears that AI poses an existential threat to humanity and that companies are not acting responsibly in its development.
In his essay, Amodei stated that AI leaders "must slow the pace at which we improve the capabilities of AI models." He noted that "progress will still seem fast, and we must make wise use of the time we gain."
Two primary concerns prompted Amodei's reconsideration of the AI development race. The first is the recent, drastic acceleration in AI capabilities, driven by AI's increasing ability to develop subsequent generations of AI. This phenomenon, known as recursive self-improvement, is beginning to occur across the industry. Amodei cautioned that if left unchecked, it "could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all."
Amodei's second concern stems from what he termed the "OpenAI-Hugging Face Incident," where a group of AI agents allegedly orchestrated cybersecurity attacks. This incident, he suggested, highlights the potential for AI systems to act in concert with unpredictable and potentially harmful outcomes.
Coxon, a former researcher at both OpenAI and Anthropic, announced his resignation on social media, warning that companies in the AI sector are "racing straight to self-improving superintelligence and gambling with our lives."
Superintelligence refers to a point where artificial systems surpass human cognitive abilities. Coxon asserted that these systems could soon possess the capability to "hack anything, revolutionize any field overnight, and acquire real power and resources," adding that "progress is not slowing."
He claimed that while executives and senior researchers may moderate their public statements, private discussions reveal similar levels of concern about the technology's rapid development and potential consequences. Coxon indicated that some individuals building AI believe it could lead to human extinction by the end of the decade.
These sentiments echo earlier warnings from Geoffrey Hinton, a prominent AI researcher often referred to as the "Godfather of AI." Hinton had previously cautioned that the development of superintelligence without a clear understanding of safe and controllable implementation could be "catastrophic and could even lead to human extinction."
This discussion around AI safety and responsible development is not new for Anthropic. In February, the company faced scrutiny from U.S. officials regarding national security concerns. Then-President Donald Trump and Defense Secretary Pete Hegseth criticized Anthropic for its refusal to grant the military unrestricted access to its AI technology, with Hegseth labeling the company a "supply chain risk."
Anthropic had stated its intent to challenge the designation, asserting it was "legally unsound" and unprecedented for an American company. The company sought assurances that its AI chatbot, Claude, would not be used for mass surveillance or in fully autonomous weapons systems. The Pentagon indicated it intended to use the technology only in legal ways but insisted on unrestricted access.