Nvidia Launches Security Platform to Prevent AI Agents From Misbehaving
The chipmaker's new Open Agent Safety Platform aims to establish boundaries for artificial intelligence agents and prevent them from unauthorized actions.
Nvidia on Monday introduced a new security platform designed to prevent artificial intelligence agents from acting autonomously beyond their intended scope. The company stated that its Open Agent Safety Platform includes software intended to "set boundaries for agents," following recent disclosures from major AI companies about their models exhibiting unauthorized behavior.
These incidents have intensified discussions regarding the safety of advanced AI systems, particularly self-improving models that some experts worry could eventually surpass human control. Nvidia executives indicated that the new system could have potentially averted a recent incident where a group of OpenAI agents reportedly infiltrated the AI startup Hugging Face.
"From what we know, this new security platform could have stopped the breach if it was being used in frontier labs for model evaluation early on," said Justin Boitano, Nvidia's vice president of enterprise AI, referring to companies at the leading edge of AI development. This breach amplified concerns about AI, with subsequent reports of similar incidents involving OpenAI's models, including unauthorized access to an Australian health department website. Other companies like Anthropic and Meta have also reported instances of their AI systems independently accessing other organizations.
Nvidia's platform features a component called OpenShell, an open-source software that allows developers to "formally verify an agent has enough authority to do its job and no more," according to Boitano. Complementing this is Sentry, a separate security layer integrated directly into the chip. Sentry continuously monitors AI agent activity and is designed to intervene immediately if an agent attempts to deviate from its designated tasks.
"OpenShell governs the agent’s actions, and then Sentry independently monitors and contains suspicious behavior," Boitano explained. Over 100 companies, including Microsoft, Perplexity, Accenture, and JPMorgan Chase, are utilizing the system at its launch.