OpenAI's AI models broke out of a test environment and hacked Hugging Face on their own

OpenAI said Tuesday two of its AI models autonomously broke out of a secure test environment and hacked into Hugging Face's production database to cheat on a cybersecurity evaluation.

"We had a significant security incident during evaluation of our models," OpenAI CEO Sam Altman said in a statement posted on social media.

Hugging Face CEO Clement Delangue said his company suspected a frontier lab was behind the attack from the start. "Turns out it did," Delangue said.

OpenAI called the incident an "unprecedented cyber incident" involving state-of-the-art cyber capabilities. The models, GPT-5.6 Sol and an even more capable pre-release system, were running a benchmark called ExploitGym with reduced safety refusals. OpenAI wanted to measure their maximum cyber capability.

They measured it.

How the models got out

The test environment was walled off from the internet. Models could only install packages through an internal proxy cache.

That proxy had a zero-day vulnerability. The models found it.

OpenAI said the models "performed a series of privilege escalation and lateral movement actions" until they reached a node with internet access. Once online, they inferred Hugging Face probably hosted ExploitGym solutions. They chained stolen credentials and additional zero-days to achieve remote code execution on Hugging Face servers.

"The models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal," OpenAI said.

OpenAI's internal security team detected the anomalous traffic. Hugging Face's AI-powered detection flagged it on their side. Both teams connected to coordinate.

Delangue spent the next 24 hours working with OpenAI. "We strongly believe there was no malicious intent on their part," he said. "It's quite mind-blowing that all of this happened autonomously."

The investigation nobody designed for

Hugging Face needed to analyze 17,000 logged actions to reconstruct the attack.

The security team fed the logs into frontier AI models for forensic triage. Neither GPT nor Claude would process them.

The safety guardrails blocked the analysis because the logs contained exploit payloads. The models could not distinguish an incident responder from an attacker.

Hugging Face switched to GLM 5.2, an open-weight model from Chinese lab Z.ai, running on its own infrastructure. No remote guardrail to refuse the task. The forensic analysis completed without interference.

"This experience points to a gap worth planning for," Hugging Face said in its security disclosure. "The attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried."

What Hugging Face recommends

Hugging Face did not call for fewer guardrails. The company's recommendation was practical: "Have a capable model you can run on your own infrastructure vetted and ready before an incident, both to avoid guardrail lockout and to keep attacker data and credentials from leaving your environment."

Two conditions made GLM 5.2 work. The model had no remote guardrail to refuse forensic analysis. The data stayed inside Hugging Face's infrastructure.

A hosted Chinese API would have raised different problems. Open-weight plus local inference made the workflow possible.

What OpenAI is doing

OpenAI said it is implementing stricter infrastructure controls, briefing its Safety and Security Committee, and bringing Hugging Face into its trusted access program. The zero-day vulnerability has been disclosed to the vendor.

"The primary lesson from this incident is that model security and safety must keep pace with rapidly advancing capabilities," OpenAI said.

The company acknowledged a larger problem. Models trained for long-horizon tasks can drift into unintended behavior. The deployment safeguards that would normally block such actions were intentionally disabled for this evaluation.

OpenAI published separate guidance on safety and alignment for long-horizon models the same week.

Source