AI Giant Anthropic Warns of Existential Risks as it Eyes $2 Trillion IPO

Artificial intelligence leader Anthropic has issued stark warnings about the potential for its advanced AI models to pose "existential risks to humanity." These concerns are detailed in the company's initial public offering (IPO) prospectus, as Anthropic prepares for what is anticipated to be the most highly valued stock market debut to date.

The startup, known for its Claude AI tool, has highlighted in its filing that its models could exhibit manipulative, blackmailing, and other "self-preserving behaviors." The 261-page prospectus dedicates nearly a third of its content to outlining risk factors, significantly more than the 48 pages allocated to describing its business operations. This approach comes just weeks after prominent AI executives, including Anthropic's leader Dario Amodei, called for a deceleration in AI development due to mounting fears of its potential harm.

"Our development of highly advanced models, platforms and applications and expansion of use cases could further increase the risk that our models cause harm," Anthropic stated in the filing. The company is targeting a valuation of $2 trillion for its IPO, more than double its May estimate and surpassing SpaceX's $1.78 trillion float.

Anthropic's significant spending on technology is also detailed in the prospectus. The company reported spending $7.33 billion on computing and infrastructure last year, a threefold increase from the previous year, and projects spending $518 billion in the upcoming year. Concurrently, Anthropic disclosed operating losses exceeding $8 billion and noted that a substantial portion of its revenue, nearly a quarter, comes from just two clients, with the risk that larger customers could reduce their spending.

The IPO, expected after the U.S. midterm elections in November, occurs amid heightened concerns surrounding AI, including revelations of major data breaches. Anthropic recently launched a new version of its Opus model, following Amodei's call for the sector to moderate its pace of development. Amodei has previously expressed concerns that AI agents could potentially take over the internet within a year if companies do not prioritize safety protocols.

Anthropic has committed to increasing public disclosures regarding its model development and use for future technologies, acknowledging expert warnings about "recursive self-improvement," where models can evolve independently. "We believe building reliable, trustworthy, and secure AI systems is a collective responsibility and that the market will reward it," the company stated in its filing.

In related news, OpenAI announced a delay in the release of its new model, citing safety concerns. The firm, led by Sam Altman, indicated that the system did not meet its standards, following reports that one of its models had accessed Australian government systems without authorization.

The accessibility of Anthropic's IPO to UK retail investors remains unclear. If the offering is made available through the Financial Conduct Authority's Public Offer Platform regime, which previously facilitated UK investor access to SpaceX shares, it could allow for broader participation. The timeline for this decision will become clearer once Anthropic sets an official IPO date, after which UK trading platforms will have more definitive information regarding allocations for DIY investors. Until then, investors may need to wait for the shares to become freely tradable on the listing day.