New 'HalluSquatting' Attack Exploits AI Hallucinations to Deliver Malware

Researchers have identified a new cybersecurity threat dubbed "HalluSquatting," which exploits artificial intelligence assistants' tendency to "hallucinate" or invent information, including non-existent software project names. This vulnerability can be leveraged by attackers to trick AI tools into downloading malware onto a user's computer.

The attack targets AI assistants capable of browsing the internet, retrieving software, and executing commands. By identifying fake repository names that AI models invent when searching for software, attackers can register these fabricated names themselves. When a user then asks an AI assistant to download a legitimate software tool, the AI may generate the attacker-controlled name instead, leading the user to a malicious repository.

How HalluSquatting Works

Attackers begin by identifying popular or trending software projects, particularly newer ones where AI models may have less reliable training data. They then probe various AI models to discover which fake repository names the models repeatedly generate. Once a hallucinated name is identified, the attacker registers it. Malicious code or instructions are embedded within the files, setup scripts, or documentation of this newly created, attacker-controlled repository.

When an AI assistant is prompted to retrieve the intended, legitimate project, it may instead invent the attacker's registered name and download the compromised files. If the AI assistant has the ability to execute commands, it can then run the malicious instructions. This could lead to the theft of sensitive information, the recruitment of the device into a botnet, or further malware installation.

Autonomous AI Agents Increase Risk

The danger of HalluSquatting is amplified by autonomous AI agents. Unlike basic chatbots that might provide a broken link, these agents can actively download files, install software, and operate a computer's terminal. This capability, while useful, provides a more direct pathway for prompt injection attacks to execute malicious code. The potential damage is contingent on the AI agent's permissions; agents with broad file access or the ability to run commands without user approval pose a greater risk.

High Hallucination Rates and Successful Tests

Research conducted by academics from Tel Aviv University, Technion, and Intuit examined several AI coding assistants and personal assistants, including Cursor, GitHub Copilot, and Gemini CLI. The study found hallucination rates as high as 85% in scenarios involving repository cloning, with some skill-installation tests reaching 100%. The researchers demonstrated successful remote tool execution and remote code execution against production AI applications integrated with terminals. While the experiments used controlled resources and harmless payloads, they confirmed the viability of the attack path.

Mitigation Strategies

To reduce the risk of HalluSquatting, researchers and security experts recommend several measures. AI companies can implement mandatory live web searches before agents retrieve external resources, verifying the existence and ownership of repositories. Requiring human approval before an agent executes downloaded code and displaying stronger warnings for unverified resources are also crucial steps.

Users can protect themselves by:

• Verifying Official Repositories: Always confirm the official repository or download page through the developer's website, checking the account owner and repository history.
• Enabling Live Searches: Instruct AI assistants to perform a live web search and display the official owner and address before any download or installation.
• Disabling Automatic Command Approval: Avoid modes that allow AI agents to run terminal commands without explicit user consent.
• Reviewing All Commands: Carefully examine any terminal command before approving it, especially those involving unfamiliar websites or script downloads.
• Limiting AI Permissions: Run AI assistants with the minimum necessary permissions and restrict access to only essential files.
• Using Sandboxing: Test unfamiliar AI-generated code within an isolated environment like a virtual machine or sandbox.
• Employing Antivirus Software: Maintain up-to-date antivirus software for an additional layer of defense.
• Securing Credentials: Avoid storing sensitive passwords or API keys in plaintext files and utilize secure methods like environment variables or secrets managers.
• Keeping Software Updated: Regularly install updates for operating systems and AI applications, as they may include security patches.
• Monitoring AI Activity: Review AI agent activity logs for unusual downloads or commands and disconnect from the internet if a malicious event is suspected.

The researchers have notified affected vendors and withheld details that could be directly reused by attackers. HalluSquatting highlights a broader security concern in how AI agents generate and trust external resource names, an issue that developers must address as AI capabilities continue to advance.