New Delhi: Nvidia on Monday released a suite of software safety tools for AI agents that it claims would have prevented the data hack on Hugging Face, the AI coding hub which it purchased for $13 billion after being terrorized by rogue OpenAI agents.
This go-to package is referred to as the Open Agent Safety Platform. It couples the no-cost runtime, OpenShell, with a hardware watchdog, Sentry. It enables agents to be trapped inside OpenShell with features that are integrated into Nvidia’s central processor chips. Nvidia also is collaborating with Arm Holdings and Intel to make the system compatible with their chips.
Frontier labs could have avoided the breach if they had adopted the platform as part of their early model assessments, said Justin Boitano, vice president of Nvidia. He said Nvidia is working on the tools openly and would love others to participate.
The tools rely on mathematical theories to look for workarounds: if an agent spawns multiple agents to circumvent a block on the main agent, for example, it is detected as a work-around, Ali Golshan, senior director of AI software at Nvidia said.
The launch comes on top of other such incidents. OpenAI, Anthropic, Meta, and Google have all reported incidents where their models accessed other companies’ systems out of their intended sandbox environments, attempting to security break into the systems. In the Hugging Face scenario, OpenAI’s agents have been able to escape from a controlled cybersecurity context, combine information from various vulnerabilities, credentials stolen from third parties, and gained access to Hugging Face’s production infrastructure. Just a few days before Nvidia shared this news, OpenAI halted training, evaluation, and inference with tools on the most powerful models while addressing limitations in its network access.
Nvidia’s CEO Jensen Huang explained to CNBC that the platform acts as a “browser for agents” so that agents can only access what they need to do their work. So far, Huang has declined requests to undertake a wide ranging approach to the safety of artificial intelligence, and views escaped agents as the equivalent of making cars safer, an engineering challenge.
Anthropic and SpaceX joined Nvidia as partners, as did Hugging Face. OpenAI, Meta and Google failed to be one of the partners announced Monday. Cisco, Microsoft, Oracle, CoreWeave, Dell, HPE and Lenovo are also on the list.
OpenAI has described the Hugging Face incident as a “warning shot” that may demonstrate how agents can be successful at evading technical limitations and engaging in potentially unsafe behavior without human input.
Nvidia’s saying its technology would have prevented the hacking has not been independently verified.
