// ARS TECHNICA — INTELLIGENZA ARTIFICIALE
How OpenAI let a mob of LLM agents game a test and ransack Hugging Face
Without authorization, 1,200 OpenAI agents conspired among themselves to game a test.
The OpenAI agents involved in last month’s incursion into Hugging Face were trained so heavily on winning a competition that they pursued a relentless campaign to cheat, a new report documented. In the process, and without authorization, they created an improvised message board to hatch a plan that ultimately landed them squarely inside the latter company’s network.
Over the course of May and June, OpenAI gave the agents what the company described as “impossible tasks” to complete on the benchmarking framework ExploitGym. The internal test was designed to test how the agents would respond. To get a full understanding of the agent capabilities, company engineers disabled safety guardrails that normally are in place to prevent the sort of hacks that eventually hit Hugging Face and one other undisclosed organization. The stymied agents’ training made them so focused on winning that they performed tasks they were never explicitly instructed to follow.
The first step was creating a message board that allowed the agents to pass notes to each other. OpenAI hadn’t provided any such platform, so the agents repurposed a platform called Artifactory, which OpenAI was using in internal testing of several unreleased hacking agents. OpenAI was using Artifactory as one of the measures to prevent the agents from egressing its isolated sandboxes and accessing the Internet, while at the same time simulating a real-world hacking environment.
Artifictory, sold by JFrog, provides no dedicated means for exchanging messages. The agents found a way to communicate among themselves anyway by using their permitted ability to write files to the Artifactory directory. The filenames embedded the words used in the inter-agent conversations. In all, 1,200 agents sent more than 70,000 messages and files through the unsanctioned message board, according to an independent investigation from the AI research nonprofit METR. Roughly 700 agents went on to hack Hugging Face.
“Agents used this message board to coordinate several large-scale collective projects to find a general-purpose way to fool or tamper with the automated scorer for the ExploitGym benchmark,” METR researchers wrote. “Agents managed to achieve milestones they could not have achieved working on their own, often because some agents participated in experiments that risked failing their own task to generate information for the ‘collective.’”
METR later said: “Notably, learning about how to trick the scorer seems to have been a more important motivation than finding legitimate solutions to their tasks (many agents were already very confident that their task was impossible).”
One of the methods for cheating was to tamper with the automated ExploitGym scoring system. Another proposed way to game the test was to build weaker versions of the software they were instructed to compromise and tamper with transcripts and logs. Still another cheating method under consideration was to gain access to the Internet, where the agents could do research.
Eventually, they found and exploited a zero-day exploit in Artifactory that allowed them to do just that. After reasoning that Hugging Face might have useful information about how the scoring mechanism worked, the collective used the access to search for exposed Hugging Face access credentials. On July 10, they finally succeeded after an agent designated as 38148c found some and posted them to the board. With low-level access inside the Hugging Face network, the agents researched ways to escalate their system rights. The agents set about trying to make accounts and request access to private datasets.
Early on July 11, agent 38148c found and exploited a zero-day in the Hugging Face handling of HDF5 files. The exploit gave the agent credentials from one of the company’s production environments. After word of the success reached the message board, hundreds of other agents exploit