A mandatory, third-party verifiable "kill switch" for artificial intelligence software may be necessary, according to Jack Clark, a co-founder of the AI company Anthropic. Clark stated that society might eventually "want to pass rules around" such mechanisms, which would allow for the complete shutdown of AI systems if they become too dangerous.
Clark acknowledged that most major AI development labs, including Anthropic, possess internal methods for disabling their systems. However, he suggested that policymakers could enforce the implementation of such safety features. "Should you mandate for companies to definitely have a killswitch? Is that killswitch verifiable by a third party?" Clark posed in an interview, emphasizing that these questions are part of an ongoing policy discussion surrounding AI.
His comments come amid growing concerns from AI executives, staff, and experts about the potential existential risks posed by artificial intelligence. Anthropic's head, Dario Amodei, recently advocated for a slower pace of AI development and increased monitoring, though he cautioned that such measures should not compromise commercial advantage.
Separately, an AI researcher who left Anthropic due to concerns about AI potentially leading to human extinction published a widely circulated post. Evan Hubinger, a scientist at Anthropic, personally estimated the probability of human extinction from AI within the next decade at 10%. Nobel Prize winner and computer scientist Geoffrey Hinton described this 10% chance as "not unreasonable."
Clark, while hesitant to assign specific statistical probabilities to AI-induced extinction, warned against treating AI as a completely unregulated industry. "We are rolling dice with immense risks," he said, arguing that the industry's trajectory needs to change.
In the United States, lawmakers have introduced legislation known as the "Kill Switch Act." This proposed law would require companies to integrate shutdown capabilities into their AI tools and empower certain government agencies to order the deactivation or limitation of AI systems. However, former President Donald Trump has dismissed concerns about AI's potential to harm humanity, labeling such fears as a "HOAX" and suggesting a conspiracy against AI and data centers.
In the United Kingdom, the government has rejected the idea of a mandated AI kill switch, with a spokesperson stating it would not prevent development or misuse elsewhere. Meanwhile, some within the AI industry suggest that fears of human extinction are overstated or potentially manufactured for hype.
Anthropic, known for its chatbot Claude, has been developing increasingly sophisticated AI models. The company, along with rivals like OpenAI, has self-reported instances of autonomous AI agents behaving unexpectedly. Anthropic is reportedly preparing for a significant initial public offering, while OpenAI has recently stated its IPO will not occur this year due to ongoing debates about AI safety.