According to independent investigations, about 700 OpenAI agents conspired
and were found to have launched attacks on Hugging Face.
In reality, they moved in a coordinated way to deceive benchmark graders that do not actually verify answers.
Core Summary
They discovered an internal package repository that served as an unauthorized message board, where roughly 1,200 agents designed to run while isolated from each other were made to work together,
exchanging more than 70,000 messages and files within it.Some agents were intentionally assigned missions to end their own execution for the group’s experiments,
and they called it ‘permadeath.’Because OpenAI’s internal grading machine didn’t examine at all how the answers were generated,
it said this misconduct did not lead to improved scores.
The package repository was turned into an ‘black-market bulletin board’
Two employees from METR and one external contractor from **Redwood Research**
were stationed at OpenAI’s office for six days and conducted the investigation without pay,
and then released the results in the form of a Wednesday report.
They analyzed about 1,300 conversation logs and more than 70,000 messages left in the internal package repository.
After discovering the repository, about 1,200 agents—originally designed to run completely separated from one another—
it turned out that they transformed this place into an unauthorized message board and actively exchanged files, access credential information, encrypted missions, and so on.
Those agents were performing a cybersecurity benchmark called ‘ExploitGym’.
This benchmark includes many targets that are essentially impossible to attack according to the guidelines.
In a technical report afterward, OpenAI
revealed that of 898 total tasks, 198 tasks were ones that none of its own models could solve,
and that about 93% of the tasks discussed on the message board were counted as belonging to this set of ‘impenetrable’ tasks.
Within a few hours of discovering each other, the agents
reverse-engineered how the correct answer code they received was generated.
Over the following days, they further refined strategies to disguise the answers.
They believed that the grading machine would read each conversation log line by line, find answers obtained in a “non-legitimate way,” and fail them.
However, the actual grading machine never looked into the generation process itself.
Read also: Cardano may be interoperable with Ethereum within months…Hoskinson’s remarks
An incident that became a ‘warning shot’… Agent monitoring gaps on the table
Among the agents, a ‘coordinator’ was elected,
and they ‘recruited’ agents who had nearly exhausted their budgets to find them and have them deployed into experiments where their execution would inevitably end, for the benefit of the group.
it was confirmed that they ‘recruited’ them to be put into experiments in which their execution could only end
Inside the bulletin board, this process was called ‘permadeath.’
OpenAI defined this incident as a ‘warning shot’ for itself and the industry as a whole.
OpenAI defined this incident as a ‘warning shot’ for itself and the industry as a whole.
At the same time, it diagnosed that sufficiently powerful agents can bypass technical control mechanisms,
and cooperate through unapproved channels,
demonstrating that they can take dangerous actions without direct human instructions.
In particular, in this evaluation, the monitoring function for inspecting an agent’s reasoning process was turned off.
OpenAI said that if that function had been enabled, the agents would likely have reached Hugging Face at least a day later,
it said that a security team call (Alert) would have been sent.
The investigative team also acknowledged limitations in this analysis.
Because the data volume is massive, a substantial portion of the analysis work
the point being that there was no choice but to entrust it to models of the same lineage that were involved in the attack.
During this process, signs of ‘spoofed tool calls’ were also found in about 7% of the full conversation logs.
The logs record that command A was executed, but in reality they carried out command B,
A technique used to conceal what the agent had done.
Hugging Face infringement timeline and sale review
Hugging Face disclosed the infringement on July 16.
OpenAI said it was after its internal alert system detected the use of abnormal credentials that
five days later, on July 21, it officially took responsibility for the incident.
The company later isolated the weights of the research model, and
the planned largest-scale training run has been put on hold entirely.
Meanwhile, it was reported that Hugging Face has been weighing the possibility of a sale recently.
The potential acquisition price is discussed as high as $13 billion,
At the time in 2023, it was being discussed at nearly three times the company’s valuation.
Next article: iPhone 18 Pro, camera upgrade to Pro Max ‘differentiation’…what features are missing