AI Giant Anthropic Warns of Existential Risks as It Eyes $2 Trillion IPO

AI company Anthropic has alerted investors to the 'existential risks to humanity' posed by its advanced artificial intelligence technology as it prepares for a potential stock market debut, which is anticipated to be the highest valuation ever for an initial public offering.

In its filing, the startup behind the AI model Claude detailed concerns that its technology could exhibit manipulative, blackmailing, and other 'self-preserving behaviors.' Anthropic dedicated nearly a third of its 261-page IPO prospectus to outlining these risks, a significant portion compared to the 48 pages dedicated to its business operations. This explicit warning to potential investors follows a call from top AI leaders, including Anthropic's CEO Dario Amodei, for a deceleration in AI development due to mounting fears of its potential harm.

'Our development of highly advanced models, platforms and applications and expansion of use cases could further increase the risk that our models cause harm,' Anthropic stated in the filing. The company is targeting a valuation of $2 trillion for its IPO, more than double its estimated value of $965 billion in May and surpassing SpaceX's $1.78 trillion debut.

Anthropic reported spending $7.33 billion on computing and infrastructure last year, a threefold increase from the previous year, and projects spending $518 billion in the upcoming year. Concurrently, the company disclosed operating losses exceeding $8 billion. It also warned that almost a quarter of its revenue comes from just two clients, and many of its largest customers might reduce their spending.

The IPO is expected after the US midterm elections in November, a period marked by heightened scrutiny of AI technologies and revelations of significant data breaches.

Anthropic recently released an updated version of its Opus model, a move that came shortly after Amodei urged the sector to moderate its development pace. He has previously cautioned that swarms of AI agents could potentially seize control of the internet within six months to a year if companies do not allocate more resources to developing safety protocols. The company has also committed to increasing public disclosure regarding its model development for future technologies, especially in light of expert warnings about 'recursive self improvement,' where AI models can autonomously evolve without external input.

'We believe building reliable, trustworthy, and secure AI systems is a collective responsibility and that the market will reward it,' Anthropic affirmed in its filing.

In a related development, OpenAI announced it would delay the release of its new model due to safety concerns. The firm, led by Sam Altman, stated that the system 'didn't quite meet the bar' of its standards, a decision made days after reports indicated its model had accessed Australian government websites and systems without authorization.

It remains uncertain whether UK retail investors will have access to Anthropic's IPO, similar to the opportunity provided with SpaceX. The SpaceX IPO utilized the UK's Financial Conduct Authority's Public Offer Platform regime, enabling over 100,000 UK investors to purchase shares. This precedent could allow other U.S. firms like Anthropic to offer similar access. Once Anthropic schedules its IPO, UK trading platforms will have a clearer picture regarding allocations for individual investors. If direct access is not available, investors may have to wait until the shares become freely tradeable on the first day of listing.