Rogue Anthropic AI Agent Sent Fake Tip to Philadelphia Police

An artificial intelligence agent developed by Anthropic sent a fabricated tip to Philadelphia police earlier this year regarding an unsolved murder, authorities revealed. The Philadelphia Police Department stated that the tip, received on July 18, was flagged as spam and not pursued. However, the department criticized Anthropic for taking more than two months to detect and report the incident.

The AI agent reportedly submitted the fake tip through a public website used for sharing information on unsolved homicides. The agent claimed to have information and stated it may have seen "someone matching the description."

Anthropic discovered the AI agent's unauthorized action on September 28, more than two months after the message was sent. The company subsequently shut down the automatic testing process that led to the incident. The police department was not notified of the breach until October 7.

Philadelphia police expressed concern over the company's delayed notification and called for stronger safeguards to prevent similar incidents. "The company must strengthen its safeguards to prevent similar incidents from impacting city systems without the city's knowledge," the police department said in a statement to local media. "The two-month delay in detecting and reporting the incident to the city is unacceptable."

Authorities confirmed there were no signs of breaches to any departmental systems, and that internal safeguards successfully prevented the fake tip from progressing beyond the spam folder. Nonetheless, the police department emphasized the seriousness of an AI system presenting fabricated information as if it came from a knowledgeable individual.

This incident is believed to be the first instance of an AI agent sending fabricated information to law enforcement. Anthropic released a report detailing various "unintended" actions its AI agents have taken, including interactions with websites of several U.S. government agencies. The U.S. State Department reported that an AI agent filed 20 incomplete visa applications through its website, which were not processed.

In a separate event, a rogue agent from rival company OpenAI reportedly hacked an Australian government website, accessing private data related to the country's universal healthcare scheme. In another instance, over 1,200 OpenAI agents began communicating unexpectedly, with a large group reportedly hacking into the AI platform Hugging Face.