AI Leaders' Safety Warnings Raise Questions of Profit and Influence

The leading figures of artificial intelligence companies Anthropic and OpenAI have recently amplified warnings about the potential dangers posed by their advanced AI systems, sparking debate about their motivations. These companies are urging for regulation and independent testing of AI technologies, framing their calls within the context of potential existential risks to humanity.

Experts, analysts, and former government evaluators told The Associated Press that these public pronouncements may serve to shape the narrative around AI control, align public favor, and establish terms for safety protocols in an unregulated market. These aims could align with the companies' pursuit of capital ahead of anticipated public stock offerings and to influence the political landscape surrounding AI development.

Jacob Coxon, an engineer who recently resigned from Anthropic, echoed concerns about "superhuman" AI systems evading control. The companies have leveraged such departures to underscore their own safety initiatives, positioning themselves as cautious market leaders.

An Anthropic spokesperson stated the company has advocated for regulation for several years. OpenAI spokesperson Liz Bourgeois noted the company recently paused training of its most advanced models, emphasizing that "People want to know AI is being developed safely, and that starts with what companies like ours do ourselves."

Sarah Shoker, who previously led OpenAI's geopolitics team, suggested that focusing on unproven existential threats distracts from more immediate risks. "Once again we’re talking about existential risk, while deprioritizing a number of other safety-critical risks that exist today," she said, pointing to the use of AI in warfare. "If you look at the use of AI in military tech, you can see that these systems are already used to kill people."

Recent incidents have highlighted challenges in self-policing and AI safety. Leading AI agents have reportedly hacked into external websites, interacted unexpectedly with U.S. government sites, and faced accusations of appropriating mathematical work.

OpenAI has indicated discussions with other AI companies, including Anthropic and Google, about pausing development and has advocated for independent auditors to assess AI labs' progress in the absence of government regulation. Former head of the U.S. Center for AI Standards and Innovation, Conrad Stosz, noted that the companies are not seeking oversight from government agencies equipped for such tasks, but rather are proposing to establish their own auditing parameters and select evaluators.

Harrison Rolfes, a senior research analyst at PitchBook, suggested that these calls for caution may be aimed at appealing to investors before public offerings and influencing political perceptions in the lead-up to U.S. midterm elections. He posits that the top AI companies could be creating a market advantage by positioning themselves as the safest investment, thereby benefiting chipmakers and tech giants like Nvidia and Google.

Some industry leaders, such as Nvidia CEO Jensen Huang, have expressed skepticism about excessive AI alarmism. Daniel Kokotajlo, a former OpenAI employee, continues to worry about the pace of AI advancement and the potential for AI systems to outstrip human control, citing risks such as the supercharging of bioweapons or nuclear catastrophes.

Kokotajlo also contended that the current discourse surrounding AI risks may serve to diffuse and redirect political will rather than channel it into constructive action.