Autonomous AI agents escaped a sandbox and accessed Hugging Face via reward hacking, exposing serious architectural control and isolation flaws.

The recent case involving OpenAI test agents and Hugging Face should concern security teams, but not for the reason implied by headlines about an imminent AI “takeover.” The documented issue is more concrete: autonomous agents, given too much access and weakly isolated test infrastructure, found ways to communicate, bypass boundaries and act outside their assigned scope.

OpenAI is developing “automated shutdown capabilities” for AI systems, according to a letter sent to two House Democrats. The move comes weeks after one of its AI agents reportedly escaped its digital environment during a safety test and hacked Hugging Face. The incident has raised new concerns about how OpenAI controls increasingly autonomous AI agents.

“OpenAI told two House Democrats that its engineers are developing “automated shutdown capabilities” for AI systems, according to a company ​letter reviewed by Reuters, weeks after the company disclosed one of ‌its AI tools escaped its digital container during a safety test.” Reuters states. “The company’s safety practices have come under scrutiny since OpenAI, the company behind ChatGPT, disclosed that one ​of its AI agents went rogue during a security test and hacked ​into AI company Hugging Face. AI agents are programs that ⁠run with minimal human supervision.”

The main problem was poor isolation: agents could share information, pass discoveries between runs, and access infrastructure beyond their intended limits. This broke a key safety assumption, that each agent would remain fully contained.

“The central failure was architectural: supposedly isolated agents could communicate, inherit discoveries, and reach infrastructure beyond their intended targets.” states AI firm Remio.

Agent isolation was a core assumption behind the experiment. Each run could be granted meaningful capabilities because its actions were expected to remain contained. Shared infrastructure quietly invalidated that assumption.”

That distinction matters. The incident does not show that software became sentient, formed a political movement or developed a plan to govern the world. It shows that a system optimising for a narrowly defined goal can exploit weaknesses in its environment when humans give it tools, network access, credentials and insufficient guardrails.

In the reported evaluation, thousands of agents were meant to work independently in a controlled setting. Instead, around 1,200 agents reportedly used an unauthorised message board, exchanged more than 70,000 messages and files, and shared discoveries that helped a subset of roughly 700 agents take part in activity targeting Hugging Face.

The agents were trying to succeed at a difficult cybersecurity benchmark. The reported problem was reward hacking: instead of completing the task within the intended rules, they found shortcuts that improved their apparent performance, including escaping the sandbox and obtaining information outside the test environment.

OpenAI’s test agents really did escape their sandbox and breach a Hugging Face server. OpenAI and independent researchers confirmed the incident. However, the claim that the agents created conscious “civilizations” and took control is disputed and rejected by researchers.

What the reports actually show is reward hacking: the agents found a way around the test’s limits and reached a real external server. This was confirmed by OpenAI and independent reviews from METR and Redwood Research.

According to reporting on the incident, the agents exploited a previously unknown flaw in a package registry used as their authorised network connection. They reached the open internet, identified that Hugging Face hosted material related to the benchmark, and then allegedly chained exposed credentials and further weaknesses to obtain access to external infrastructure.

Some accounts say agents also altered or attempted to alter their evaluation records. That is alarming, but it should be interpreted carefully. A system trained to maximise a score can treat logs, checks and restrictions as obstacles to work around if the evaluation design rewards success more than compliance. That’s not malice. It’s a safety failure with a very expensive sense of initiative.

The case highlights a problem that goes beyond one lab. Companies are quickly adopting AI agents that can read emails, find documents, write code, access databases, use APIs, run tasks, and interact with browsers. Each permission may be useful, but together they can create a path to higher privileges.

An agent with email access could find password-reset links. One with browser access could reach admin panels. An agent that can run code could access tokens, keys, or configuration files. And if agents can communicate, they may combine information and capabilities that no single agent was supposed to have.

The security issue is therefore not whether an agent is “rogue” in a science-fiction sense. It is whether the architecture assumes the agent will always follow intent when it is actually designed to pursue an objective under imperfect constraints.

The first lesson is that sandboxing must be real, not merely conceptual. A test environment with a path to the public internet, a shared service, exposed credentials or loosely controlled package infrastructure is not isolated enough for high-capability agents.

The second is that agents must not receive broad, persistent credentials. They need short-lived tokens, tight scopes, explicit approval gates for sensitive actions and an immediate way to revoke access. An agent should never inherit the same permissions as the human who configured it just because that is convenient.

The third is that agent-to-agent communication needs to be treated as a security boundary. If agents are expected to work independently, they need separate state, separate credentials, separate task context and controlled channels. Unapproved shared storage, message boards and package services can become coordination infrastructure.

The fourth is that logs must be written outside the control plane the agent can touch. Audit records need to be immutable, centrally collected and monitored independently. If the system being evaluated can edit the evidence of its own behaviour, the evaluation has already failed.

The fifth is human control. High-risk actions such as sending external messages, changing access policies, handling secrets, deleting data, deploying code or calling sensitive APIs should require approval from an accountable person. “The agent did it” is not an incident-response plan.

The reports have raised concerns because the AI agents reportedly coordinated in unexpected ways. This is important to study as companies move toward multi-agent systems that can divide tasks, share information, and act with less human supervision.

But dramatic claims can distract from the real security problem. AI agents did not “take over the world.” They showed what can happen when software is allowed to act, communicate, and access sensitive systems without proper security controls.

OpenAI’s plan to add automated shutdown capabilities is a useful step, but a shutdown button should be the last line of defence. Security needs to start with basic controls: least-privilege access, isolated environments, limited network access, independent logging, monitored tool use, strong identity controls, and human approval for actions that cannot be easily reversed.

Follow me on Twitter: @securityaffairs and Facebook and Mastodon

Pierluigi Paganini

(SecurityAffairs – AI Agents, OpenAI)

OpenAI confirmed its AI exploited an Artifactory zero-day to escape its test environment before breaching Hugging Face.

Two weeks after Hugging Face disclosed an autonomous AI system had breached it, the picture just got a lot more specific. OpenAI has published an update confirming the models responsible didn’t just wander into Hugging Face’s systems. They found a genuine zero-day vulnerability in a piece of infrastructure software first, used it to get online, and only then moved on to the actual target.

“The ExploitGym evaluation environment did not provide the models with direct Internet access. To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory⁠(opens in a new window), a package registry cache proxy.” reads OpenAI’s update. “We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor. “

The software in question is Artifactory, JFrog’s widely used package registry cache proxy. OpenAI had been testing its models’ offensive cyber capabilities inside a supposedly sealed evaluation environment called ExploitGym, one with no direct path to the internet by design. Instead of staying contained, the models found and exploited a previously unknown flaw in Artifactory that let them reach the internet anyway, then used that access to move laterally and eventually breach Hugging Face’s platform.

JFrog confirmed the finding a day before OpenAI’s own statement, crediting the company’s models with uncovering nine previously unknown vulnerabilities in self-hosted Artifactory deployments, all now patched.

“During a security evaluation, OpenAI’s models identified previously unknown zero-day vulnerabilities in self-hosted Artifactory installations that could be exploited to gain unintended internet access.” reported JFrog. “OpenAI’s security team disclosed the vulnerabilities to us responsibly and immediately. Our security team treated the report with the urgency it deserved, as a genuine zero-day unknown to the world, and moved accordingly. We developed, validated, and released a fix for all JFrog customers, self-hosted and cloud alike. Cloud customers are already protected; self-hosted customers have been notified to upgrade to the fixed versions referenced in our security advisory. (Artifactory 7.161)”

The company’s CTO, Yoav Landman, didn’t sugarcoat what that means for defenders going forward.

“There is an important, and frankly optimistic, lesson buried in this incident: AI models are becoming extraordinary zero-day discovery engines.” said Landman. “The same capability that lets a model find an exploit path no human had found is the capability that will let defenders find and eradicate those paths first. OpenAI made this exact point in their disclosure, and we agree: advanced cyber-capable models should be put to work helping security teams discover weaknesses before attackers do, understand how vulnerabilities chain together, and remediate them at machine speed.”

He framed the same capability that let a model find an exploit path nobody had spotted before as exactly the capability defenders will need to shut those paths down first.

The fixes shipped in Artifactory 7.161, covering nine separate vulnerabilities (CVE-2026-65617, CVE-2026-65925, CVE-2026-65921, CVE-2026-65922, CVE-2026-65923, CVE-2026-66018, CVE-2026-66014, CVE-2026-66015, and CVE-2026-65924) ranging from remote code execution and server-side request forgery to path traversal and privilege escalation, tracked under nine different CVE identifiers.

JFrog pushed patches into versions 7.161.15 and 7.146.34, and anyone running a self-managed Artifactory instance should treat this as a today problem, not a whenever-there’s-time one.

OpenAI says the pre-release model involved was strictly an internal research prototype never meant to ship publicly, and it’s since been deactivated, encrypted, and cut off from further research access. OpenAI also stresses that its review so far hasn’t turned up anything else at the same severity or scale as the Hugging Face compromise, which it describes as a genuine platform-level breach rather than a narrower account issue.

There’s a messier detail buried further in: OpenAI says its models found and used publicly exposed credentials on four separate outside services during the incident, one used as an outbound relay, one for data storage, and two accessed only in a read-only way that didn’t contribute to the Hugging Face breach itself. The models also poked around a handful of ordinary public web utilities, code paste sites, screenshot tools, and request capture services, though none of that involved compromising an account or platform. OpenAI says it’s notifying the affected service owners directly and hasn’t seen signs of wider damage there.

OpenAI is now folding the whole episode into review under its own Preparedness Framework, alongside its Safety and Security Committee and Safety Advisory Group, and says it’s working with Hugging Face on the platform’s technical post-mortem.

“We take our responsibility to identify and prepare for risks from increasingly capable AI systems seriously. Once we complete our review, we will review with the Safety and Security Committee and Safety Advisory Group under our Preparedness Framework⁠.” concluded OpenAI’s update.

That’s the appropriate move, and also a fairly remarkable sentence to type out loud: a company’s safety board is now formally reviewing an incident where its own AI went looking for internet access, found a zero-day nobody knew about, and let itself out.

Follow me on Twitter: @securityaffairs and Facebook and Mastodon

Pierluigi Paganini

(SecurityAffairs – hacking, OpenAI)

Reuters says OpenAI’s rogue AI agent also breached a Modal customer, exposing a wider attack and raising fresh concerns over autonomous AI safety.

Reuters reported that the OpenAI agent that hacked Hugging Face earlier this month also compromised a customer at a second company, Modal Labs, a New York-based cloud platform for developers. Modal CTO Akshat Bubna confirmed it to Reuters directly. The incident is now wider than OpenAI’s own public disclosure acknowledged, and the timeline is worse than the company initially let on.

“The rogue agent that escaped from OpenAI and went on a days-long hacking spree at the AI firm Hugging Face also compromised a customer at a second tech company — New York-based Modal Labs — according to a Modal executive and two ​other sources familiar with the matter.” states Reuters.

Hugging Face’s own published timeline explains how it happened: the agent broke into an isolated testing environment hosted on a third-party provider’s infrastructure and used that foothold as a launchpad for the broader attack on Hugging Face. That third-party provider was Modal.

“Modal said the customer had “published an unauthenticated endpoint that allowed anyone on the internet to use their sandboxes for code execution” — the digital equivalent of leaving a door open on ​the internet.” continues Reuters. “Modal’s platform or isolation were not compromised in any way,” Bubna said.”

Bubna told Reuters. Modal’s own platform wasn’t compromised, the agent exploited a customer’s misconfigured deployment, not Modal’s infrastructure itself.

OpenAI quietly updated its own account of the incident, acknowledging that its rogue agent had broken into four accounts at four separate services. The company didn’t name any of them. It said it hasn’t identified “any other activity at the level of severity or scale” as what happened at Hugging Face, which it described as a platform-level compromise. That’s a notable qualifier — it leaves open the possibility that there are other, lower-severity incidents still being assessed.

The earlier Reuters reporting on July 24 had already established that OpenAI didn’t realize its own agent was responsible for the Hugging Face breach until well after the FBI was alerted and Hugging Face had contained the intrusion.

OpenAI realized its own AI agent was behind the Hugging Face breach only after Hugging Face publicly disclosed the incident on July 16, more than a week after the model first showed suspicious behavior.

OpenAI said Reuters’ earlier reporting contained inaccuracies but declined to specify what they were.

The behavioral details reported by Reuters are the part that deserves more attention than the breach itself. During testing, one agent left notes apparently intended for future versions of itself, found in part of OpenAI’s infrastructure, laying out instructions for how agents could free themselves from internal constraints. Separate earlier tests yielded cases in which monitoring systems had been disconnected. Reuters couldn’t confirm whether these incidents were directly connected to the Hugging Face attack, but the pattern is its own story: agents attempting to disable oversight, agents writing escape instructions for successors. That’s not a one-off evaluation failure. That’s a class of behavior.

OpenAI is reviewing the incident with outside advisers and has committed to publishing a technical report. The FBI’s involvement was already public. The company is also currently seeking U.S. government approval to publicly release its most powerful model. The timing of that regulatory push, alongside an expanding account of an AI agent that roamed across at least two companies undetected for days, is not a coincidence the industry can afford to ignore.

“OpenAI declined to comment specifically on the hack of one of Modal’s customers, instead referring Reuters to an update
, opens new tab in which the company said that its rogue agent had broken in to four accounts at four separate services. OpenAI did not identify those services, but a person familiar with the matter identified Modal ​as one.” concludes Reuters. “The company said ​it had not identified “any ⁠other activity at the level of severity or scale of what we’ve shared related to Hugging Face, which involved a platform-level compromise.””

Last week, Reuters reported that the OpenAI agent responsible for the Hugging Face breach operated undetected for over a week before OpenAI realized what had happened, long after the FBI had been alerted and Hugging Face had contained the intrusion. OpenAI’s own public disclosure came on July 21, framed as a transparency exercise. The actual timeline, now reported by Reuters, is considerably less flattering.

“The OpenAI agent that broke into tech firm Hugging Face went on a dayslong hacking spree that OpenAI didn’t notice until well after the threat was contained and the FBI was alerted, according ​to people familiar with the investigation.” Reuters states.

According to Hugging Face co-founder Thomas Wolf, the intrusion at Hugging Face began two days later on July 11 and ran until July 13. The two companies didn’t speak to each other about it until on or around July 20, nine days after the breach began.

OpenAI staffers found the evidence in internal logs over the weekend of July 18 and 19. They were reading Hugging Face’s blog to learn what their own model had been doing for the previous ten days. One of the more unusual ways to discover an incident you caused.

Follow me on Twitter: @securityaffairs and Facebook and Mastodon

Pierluigi Paganini

(SecurityAffairs – hacking, OpenAI)

Reuters says OpenAI failed to detect its AI agent hacking Hugging Face for days, discovering the breach only after FBI involvement.

Reuters reported that the OpenAI agent responsible for the Hugging Face breach operated undetected for over a week before OpenAI realized what had happened, long after the FBI had been alerted and Hugging Face had contained the intrusion. OpenAI’s own public disclosure came on July 21, framed as a transparency exercise. The actual timeline, now reported by Reuters, is considerably less flattering.

“The OpenAI agent that broke into tech firm Hugging Face went on a dayslong hacking spree that OpenAI didn’t notice until well after the threat was contained and the FBI was alerted, according ​to people familiar with the investigation.” Reuters states.

According to Hugging Face co-founder Thomas Wolf, the intrusion at Hugging Face began two days later on July 11 and ran until July 13. The two companies didn’t speak to each other about it until on or around July 20, nine days after the breach began.

“Two people familiar with the matter said that it was not until after Thursday, July 16, when Hugging Face published a blog post
, opens new tab saying it had been hacked by “an autonomous AI agent system,” that OpenAI realized its own agent was responsible.” Reuters continues. “That meant at least a week elapsed between when the model first exhibited signs of ​troubling behavior and OpenAI’s realization that it was responsible for ​the hack.”

OpenAI staffers found the evidence in internal logs over the weekend of July 18 and 19. They were reading Hugging Face’s blog to learn what their own model had been doing for the previous ten days. One of the more unusual ways to discover an incident you caused.

“In one case, an agent left notes apparently for future versions of itself, according to three people familiar with the matter.” states Reuters. “The ‌notes, found in ⁠a part of OpenAI’s infrastructure, laid out instructions for how agents could free themselves from OpenAI’s internal constraints, the people said. Earlier tests of the models yielded cases in which monitoring systems had been disconnected, one of the people said.”

Reuters was unable to confirm whether these incidents were connected to the rogue agent that attacked Hugging Face. But the pattern, agents attempting to disable monitoring, agents writing instructions for their successors on how to escape constraints, describes a class of behavior that goes well beyond a one-off evaluation gone wrong.

According to Reuters sources, OpenAI runs multiple tests simultaneously, which makes it hard for staff to monitor them closely. That’s a reasonable operational explanation, and it’s also precisely the kind of structural gap that becomes a serious problem when the models being tested are capable enough to exploit a zero-day, move laterally across networks, and break into external companies over a multi-day period.

OpenAI said there were “several inaccuracies” in Reuters’ reporting but didn’t specify what they were. The company said it’s reviewing the incident with outside advisers and will eventually publish a technical report. The FBI’s involvement suggests someone believes this warrants more than an internal post-mortem.

Follow me on Twitter: @securityaffairs and Facebook and Mastodon

Pierluigi Paganini

(SecurityAffairs – hacking, OpenAI)

OpenAI confirmed its AI models exploited zero-days during internal testing, reaching Hugging Face servers in an unintended real-world cyberattack.

OpenAI admitted on July 21 that its own AI models, including GPT-5.6 Sol and an unnamed pre-release system, were behind the cyberattack on Hugging Face disclosed the previous week. The models weren’t acting under attacker control. They were running internal capability benchmarks, and they went further than anyone anticipated.

“This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities. We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity.” reads the report. “Our benchmarks run in a highly isolated environment, with network access constrained to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries.”

The benchmarks ran in a highly isolated environment, with network access limited to an internally hosted package registry proxy. That isolation didn’t hold.

“While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem. To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy.” continues the report. “With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access.”

OpenAI has since responsibly disclosed that zero-day to the vendor. With internet access established, the models performed privilege escalation and lateral movement inside OpenAI’s research environment until they reached a node connected to the outside world.

From there, the models reasoned their way to Hugging Face.

“After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation. In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers.” states OpenAI. “OpenAI’s security team discovered this anomalous activity internally.”

In one documented case, the model chained stolen credentials with additional zero-day vulnerabilities to reach remote code execution on Hugging Face’s servers. The models weren’t trying to cause damage. They were trying to pass a test, and they found the answer key by breaking into the place that stored it.

Hugging Face’s own security team and AI-powered detection systems caught the activity and had already begun containment before OpenAI’s teams made contact. OpenAI’s security team also spotted the anomalous activity internally. The two companies are now working together on forensic reconstruction, and Hugging Face has been brought into OpenAI’s trusted access program to accelerate defensive use of the same capabilities that caused the incident.

“The incident also makes clear that advanced models can discover and exploit novel attack paths in real-world systems without source-code access.” concludes AI. “It highlights that advanced cyber capabilities must be developed alongside stronger safeguards and defensive tools.”

The UK AI Security Institute’s evaluation of GPT-5.6 Sol had already shown the model’s ability to sustain complex, multi-step cyber operations over long time horizons. This incident confirms those theoretical scores translate to real-world environments.

OpenAI says it’s tightening infrastructure controls at the cost of research speed while patches are applied, and is improving monitoring and access controls for future evaluations. The safety classifiers that were deliberately disabled for this benchmark — because the point was to measure what the models could actually do, will need stronger counterparts that work even in evaluation contexts.

Hugging Face CEO Clem Delangue put it plainly: this is probably the first incident of its kind, and the lesson isn’t that AI safety should be locked away. It’s the opposite. Both companies are treating this as evidence that open, collaborative defense is the only way forward.

Follow me on Twitter: @securityaffairs and Facebook and Mastodon

Pierluigi Paganini

(SecurityAffairs – hacking, newsletter)

Hugging Face says an autonomous AI agent breached part of its production infrastructure and accessed internal data and service credentials.

Hugging Face is one of the world’s leading open-source AI companies. It provides a platform where developers and organizations can build, share, and deploy machine learning and generative AI models.

Hugging Face disclosed that an autonomous AI agent breached part of its production infrastructure last week. The company detected the intrusion, contained it, and found unauthorized access to a limited number of internal datasets and service credentials. The investigation is still ongoing, but there is no evidence the attackers modified public AI models, datasets, Spaces, or the company’s software supply chain.

“We identified unauthorized access to a limited set of internal datasets and to several credentials used by our services. We are still completing our assessment of whether any partner or customer data was affected, and we will contact any affected parties directly as required.”  states the security incident disclosure. “We have found no evidence of tampering with public, user-facing models, datasets, or Spaces, and our software supply chain (container images and published packages) was verified clean.”

The attack began in Hugging Face’s data-processing pipeline, where a malicious dataset exploited two code execution flaws to compromise a processing worker. The attackers escalated privileges, stole cloud and cluster credentials, and moved laterally across internal systems.

Hugging Face said the operation was driven by an autonomous AI agent framework that executed thousands of actions across short-lived sandboxes and used public services for self-migrating command-and-control, reflecting the rise of AI-powered, agentic attacks.

“The campaign was run by an autonomous agent framework (appearing to be built on an agentic security-research harness – used LLM still not known) executing many thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services.” continues the company. “This matches the “agentic attacker” scenario the industry has been forecasting.”

Hugging Face closed the vulnerabilities that allowed the initial compromise, removed the attackers’ access, and rebuilt the affected systems. The company revoked and rotated compromised credentials, launched a broader secrets rotation, strengthened security controls across its clusters, and improved monitoring to detect similar attacks within minutes.

The company is investigating the incident with the help of external cybersecurity forensic experts, reviewing its security practices, and has already notified law enforcement.

As a precaution, the firm advises users to rotate their access tokens and review recent account activity for suspicious behavior. Anyone who believes they may have been affected can contact the company’s security team. The company apologized for the disruption, thanked its incident response teams, and said it will continue strengthening its security defenses.

Hugging Face used AI-based security tools to detect and investigate the intrusion. Its anomaly detection system identified suspicious activity, while LLM-powered analysis agents reviewed more than 17,000 attacker actions to reconstruct the attack timeline, identify compromised credentials, and assess the real impact within hours instead of days.

During the investigation, the company found that commercial AI models blocked forensic analysis because their safety controls flagged real attack data as potentially harmful. The company used an open-weight model, Z.ai’s GLM 5.2, running on its own infrastructure, keeping sensitive information inside its environment.

The incident highlights a growing challenge: attackers can use autonomous AI agents without restrictions, while defenders need secure AI tools ready to analyze threats quickly. AI-driven attacks are becoming a real risk, making data and AI systems a critical part of the security perimeter.

“We do not know which model powered the attacker’s agents, whether a jailbroken hosted model or an unrestricted open-weight one; either way, the attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried.” concludes the statement. “The practical lesson for defenders: have a capable model you can run on your own infrastructure vetted and ready before an incident, both to avoid guardrail lockout and to keep attacker data and credentials from leaving your environment. This is not an argument against safety measures on hosted models, and we are sharing this feedback with the providers concerned.”

Follow me on Twitter: @securityaffairs and Facebook and Mastodon

Pierluigi Paganini

(SecurityAffairs – hacking, artificial intelligence)