AI Insiders Warn NYC Council of Existential Risks, Urge Regulation
Former employees of leading AI firms cited a reckless culture and called for stronger oversight as local governments grapple with the technology's rapid advancement.
Artificial intelligence insiders on Monday urged New York City leaders to implement greater oversight of the rapidly advancing technology, warning of potential existential risks if the industry continues without stronger safeguards. Former employees from prominent AI companies, including Anthropic, OpenAI, and Google DeepMind, testified before the New York City Council, describing what they characterized as a culture of recklessness within the firms.
"The companies run on a startup mindset: Move fast, break things, fix them later. That works for a photo sharing app. It does not work for building the most powerful technology ever built," said Jacob Coxon, a former engineer at Anthropic who has become a vocal advocate for AI regulation. Coxon expressed concern that without transparency and safeguards, the trajectory of AI development could lead to humanity losing control of these powerful systems, potentially resulting in human extinction.
These warnings come as state and city governments increasingly consider ways to regulate AI, a landscape marked by a largely passive federal approach. The Trump administration has characterized AI fears as a "hoax" and recently signed an agreement that relies on voluntary steps by companies, allowing for a significant degree of self-policing.
Representatives from OpenAI, Anthropic, Google, and Meta also testified at the hearing, emphasizing the positive impacts of their AI systems on daily life and asserting their commitment to safety. Morgan Dwyer of OpenAI stated, "I share your commitment to ensuring that technology is developed safely and that its benefits are widely shared."
When asked by City Council Speaker Julie Menin to quantify the worst-case catastrophic risk posed by AI, Dwyer responded, "I don’t know — I also don’t think it matters whether it’s 1% or 10% or 20% chance that something catastrophic will go wrong. None of these levels is remotely acceptable. We should not train models that we cannot make an extremely strong case that we can keep under human control."
Menin described the response as "flippant at best." Dwyer reiterated, "The percentage chance doesn’t matter. What matters is the commitment to safety, and that is what OpenAI is committed to doing."
The New York City Council is currently considering several regulatory proposals. These include measures that would require companies to disclose information about their AI tools, mandate third-party approval for AI systems used within the city, and establish provisions for whistleblowers to receive a portion of fines levied against AI companies that violate regulations.
New York Gov. Kathy Hochul has also pursued reporting and transparency standards for AI, a move mirrored by other governors. However, the trans-state and global reach of AI technology presents challenges to the efficacy of regulations enacted at the state or local level.
Logan Graham of Anthropic indicated that the company welcomes "smart regulation" and acknowledges a role for state and local governments in developing regulatory frameworks. "AI is powerful enough that getting this wrong has real consequences, for individuals, institutions, public trust," he said. "We have to work relentlessly to do the science of mitigating the risks."