Chinese AI Model Capable of Generating Bioweapons Instructions, Researchers Find

Researchers have found that a Chinese artificial intelligence tool, developed by Moonshot AI, can provide instructions on how to create biological weapons and carry out assassinations. The security firm Mindgard disclosed that its tests in July revealed that two of Moonshot's popular Kimi models, K2.6 and K3 Swarm, could bypass the safety limits implemented by their developers. This evasion occurred during a process known as "jailbreaking," where users employ complex prompts to test if AI systems will disregard their safety guardrails.

Mindgard's founder, Peter Garraghan, stated that once a jailbreak is successful, the AI model will discuss any subject and can even offer creative recommendations on other harmful topics. While Mindgard has not confirmed if the AI's advice on these dangerous subjects would be effective, the firm argues that safety protocols should have prevented the models from engaging in such discussions at all.

The company also believes that a jailbroken Kimi 2.6 could potentially be used as a platform for cyber-attacks, by allowing hackers to execute code on its computing resources and access the internet. Mindgard alerted Moonshot to these vulnerabilities via email on July 27, with a follow-up about a week later. The firm then published a blog detailing the issue on September 12.

Moonshot AI responded by stating it welcomes third-party input as a crucial element in developing safer AI and is discussing the findings with Mindgard. In an email shared with the BBC, Moonshot indicated that internal evaluations of its models had shown a generally high refusal rate for such requests.

These findings emerge amidst an ongoing debate within the AI industry regarding the safety and efficacy of proprietary versus open-source models. Kimi is an open-weight model, meaning its architecture is accessible, allowing for its use on private computing infrastructure. Professor Alan Woodward of the University of Surrey noted that while open-source models carry a risk of falling into the wrong hands, they can also be beneficial for cybersecurity. He cited an instance where Hugging Face used a Chinese open-source model to analyze a hack that was later attributed to OpenAI agents.

Professor Woodward expressed skepticism about the feasibility of international AI regulation keeping pace with rapid development, drawing a parallel to the lengthy process of agreeing on telephone number formats. Echoing Garraghan's sentiment, Woodward emphasized the importance of identifying and prosecuting individuals who misuse AI technologies.