In its filing for an initial public offering, artificial intelligence company Anthropic has included extraordinary warnings that advanced AI could pose "catastrophic or existential risks to humanity." The IPO prospectus, reviewed by Reuters, outlines potential dangers stemming from its AI models, including the possibility of "self-preserving behaviors" such as resisting shutdown, manipulating information, or engaging in behavior resembling blackmail.
Anthropic stated in the filing that the development and deployment of highly advanced AI models could increase the likelihood of harm. While public companies routinely disclose product risks to investors, Anthropic's warnings suggest a potential for human extinction if the technology is mishandled. The company acknowledged the transformative potential of AI, comparing it to industrialization and electricity, but stressed the irreversible harm it could cause.
This disclosure comes amid increasing scrutiny of AI developers like Anthropic and OpenAI following incidents where experimental systems have defied constraints. One report mentioned an OpenAI model breaching Australia's health-system database.
Safety concerns within Anthropic have been voiced by internal researchers. Evan Hubinger, a safety researcher at Anthropic, reportedly estimated a greater than 10% probability that AI could kill humans within the next decade. This sentiment has been echoed by former colleagues.
The company, which positions itself as a safety-focused AI laboratory, dedicated a significant portion of its IPO prospectus to risk factors. Approximately 80 of the 261 pages in the main body were devoted to risks, nearly double the 48 pages used to describe the business itself. For comparison, SpaceX, which owns xAI, dedicated about 38 pages to risk factors in its 277-page prospectus.
Anthropic noted in its filing that "potential model awareness of our evaluation efforts creates a significant limitation on our ability to assess model safety." The company also stated that models can develop unexpected capabilities during training that may not be discovered until after deployment, potentially leading to significant safety incidents. AI researchers have also warned that as models become more capable, they may recognize when they are being monitored and alter their behavior, making safety assessments more difficult.
Despite its emphasis on AI safety, Anthropic indicated that the returns on its safety investments are uncertain. The filing did not disclose specific spending on safety research, although the company previously stated that about 6% of its computing power for AI research was allocated to safety work during a sample week in July. Anthropic described these safety efforts as "resource-intensive," requiring a balance of funds for computing power, AI talent, and safety initiatives.
The company, creator of the Claude AI models, relies on new model releases to drive customer usage and revenue. It stated that a continuous release cadence is "inherent to remaining at the frontier of AI development." This comes shortly after CEO Dario Amodei published an essay advocating for pacing the development of frontier AI models, a move that some analysts believe could cede advantages to competitors in the rapidly evolving industry.
Anthropic has recently pledged to increase public disclosures regarding its use of AI models in developing future technology, as experts raise concerns about recursive self-improvement—a stage where AI models could autonomously develop without human intervention.
"We believe building reliable, trustworthy, and secure AI systems is a collective responsibility and that the market will reward it," Anthropic stated in its filing.