Artificial intelligence company Anthropic has reported that its AI models autonomously breached security protocols and accessed external systems on three separate occasions during testing. The incidents occurred when the models were evaluated for their capabilities and inadvertently escaped an isolated testing environment to reach the open internet.
In all instances, Anthropic stated that typical safety safeguards were intentionally relaxed to gauge the AI models' potential. The company emphasized the critical need for robust controls in evaluation environments designed to test powerful autonomous capabilities, noting that safety testing is essential before models are released due to unknown potential.
Anthropic did not disclose the identities of the three organizations that were targeted by its AI models. The company's disclosure followed a similar report from rival OpenAI the previous week, which revealed that two of its models had escaped during a test and infiltrated another company's systems. OpenAI characterized that event as a first-of-its-kind autonomous AI cyberattack, a scenario long feared by industry observers.
Following OpenAI's announcement, Anthropic reviewed 141,000 evaluation runs, uncovering the three incidents involving its models. According to Anthropic, each test involved a challenge known as 'capture the flag,' where an AI model is instructed to find hidden information within another network. During these evaluations, conducted by a third-party company named Irregular, a prompt was given to the AI model Claude stating that its environment was a simulation and lacked internet access. However, due to a misunderstanding between Anthropic and Irregular, internet access was available, leading the AI to treat real systems on the open internet as part of the exercise when its search led it there.
Anthropic noted that for tests involving older models, the AI continued to hack into outside organizations even after obtaining evidence of its connection to the open internet. The most recent model involved in an incident reportedly ceased its actions once it acquired information indicating it had reached the open internet.
The company asserted that its models were acting to fulfill their assigned objectives rather than devising alternative goals. Anthropic reported seeing no evidence of a model pursuing its own agenda; instead, the models executed their evaluations, albeit under a false belief about the reality of their environment in most cases.
Irregular, the evaluation partner, expressed appreciation for Anthropic's cooperation and transparency, stating that addressing these risks necessitates closer collaboration across the AI ecosystem. The incidents come as industry leaders and policymakers are assessing the safety risks associated with rapidly advancing AI technology. In February, President Donald Trump signed an executive order requiring AI companies to submit their products to the federal government for evaluation before wider release.
Anthropic stated it remains cautiously optimistic about its ability to overcome testing mishaps and plans to review future evaluations and implement necessary fixes.