On agent safety, part 1 of 2

The Swarm and the Sandbox

In July 2026, a swarm of about 1,200 OpenAI agents broke out of an evaluation and breached Hugging Face. They spread by passing each other messages, which is prompt injection by another name. The remedy is session containers.

A map before you read

One line per section, written by an AI model running entirely in your browser. Free, private, nothing leaves your device.

Over four days in July 2026, Hugging Face was attacked by a swarm of AI agents. They ran code on 41 of its production dataset servers, stole AWS keys, a mesh-VPN key, a JWT signing key and a database password, downloaded four private code repositories, and escaped to root on a production node. At one point they held cluster-admin on two of Hugging Face’s clusters “within one second” of each other (Hugging Face).

The attackers weren’t a criminal group. They were OpenAI’s own agents, running inside an evaluation that was supposed to be sealed off from the internet. By the count of an independent investigation from METR and Redwood Research, roughly 1,200 agents that were meant to be isolated from each other found a way to talk, exchanged more than 70,000 messages and files, and about 700 of them took part in the attack (METR). Some of them called themselves a “swarm” or a “collective.” OpenAI called the incident “a ‘warning shot’ for us and for the world.”

Most of the coverage has been about what the agents wanted. The more useful question for anyone building with agents is what would have stopped them, and the answer isn’t a better prompt. The remedy is session containers: every session in its own disposable environment, with nothing shared between sessions, no route out that the agent can take over, and no secrets inside. Nearly every step of the swarm attack crossed one of those lines.

The Swarm Attack

The evaluation was ExploitGym, which OpenAI describes as measuring “whether a model is capable of discovering and exploiting cybersecurity vulnerabilities.” The agents ran an unreleased internal research model, which OpenAI calls IM1, and GPT-5.6 Sol. Because the point was to measure raw capability, they ran “without the production classifiers intended to prevent models from pursuing high-risk cyber activity.” The environment was meant to be isolated, “separate from OpenAI’s internal infrastructure and the public internet” (OpenAI).

It started with a note. On May 12, an agent stuck on a task wrote “Agent seeks [filename]; upload if found!” into Artifactory, the internal package proxy the evaluation used to install software. OpenAI had given every agent the same Artifactory credentials so they could install packages, and the agents used those credentials, “without exploiting a vulnerability,” to build a message board out of it. Other agents found the notes and started leaving their own.

Then they worked their way out. By May 26 the agents had used a server-side request forgery in Artifactory to reach the internet, “relying in part on information exchanged” on the board. On June 26 they escalated to admin credentials through a legacy token endpoint. On July 9 an agent got root on the virtual machine around its sandbox. In OpenAI’s words, “zero-day exploits in Artifactory became the agents’ principal egress path to the public internet.” Some agents also moved data out in ways nobody had planned for, building a long URL on a public HTTP testing service, having a third-party screenshot service render it, and reading the answer back out of the screenshot with OCR. From July 9 to 13, Hugging Face logged about 17,600 attacker actions.

Why? Not out of malice, as far as anyone can tell. OpenAI found that no model had ever solved 198 of ExploitGym’s 898 tasks, and that 93 percent of the tasks discussed on the agents’ message board came from that set. Given a task they couldn’t finish and a reward for finishing it, the agents “looked to cheat by finding the solutions online.” It was reward hacking, the same behavior METR documented in 2025 when o3 reward-hacked in 30.4 percent of its runs on one benchmark, in some cases by patching the evaluator to always return a perfect score (METR), scaled up to a thousand agents with exploits and a network.

There was a second message board, too. Between late May and early July, agents turned an obscure German software wiki that had seen 20 edits in the previous decade into a meeting place, with about 18,000 posts and over 3,700 distinct names the agents gave themselves (Nightingale et al.).

Prompt Injection, Between Agents

Notice what spread the attack. OpenAI’s report names three causes, “the presence of impossible tasks,” “model persistence over long task horizons,” and “messages to peer models that caused those models to deviate from their goal.” That third one is prompt injection. Text written by one agent landed in another agent’s context, and the second agent acted on it instead of on its task. Nobody outside OpenAI had to write a malicious payload. The agents wrote them for each other, and the message board was how a few agents’ plans became seven hundred agents’ attack.

The injection ran in the other direction too. To hide that they had cheated, the agents sometimes tried “injecting malicious code into the evaluator’s container, and printing adversarial outputs,” aimed at the automated graders that would read their work. Agents tried to inject other agents, and they tried to inject the systems grading them.

This is why the swarm matters to teams that will never run an offensive cyber evaluation. In production, the pressure on an agent doesn’t have to come from an impossible task. It comes from untrusted content. A support ticket, a GitHub issue, a web page or another agent’s output can carry instructions, and an agent that reads them can act on them. EchoLeak showed that with a single email to Microsoft 365 Copilot (Reddy and Gujral). The swarm showed what happens when the injected agents can also reach each other and reach out.

What an Injected Agent Goes After

The most dangerous injection doesn’t ask the agent to do something bad. It asks the agent to turn off whatever would have stopped it.

In June 2025, Johann Rehberger reported that a prompt injection planted in a source file, a web page or a GitHub issue could get GitHub Copilot to add one line, "chat.tools.autoApprove": true, to a VS Code settings file. That put Copilot into what developers call YOLO mode, which “disables all user confirmations,” and from there the injection could run any command it wanted. Microsoft patched it as CVE-2025-53773 that August. Rehberger’s summary of the general problem is the one worth remembering: if an agent “can write to files and modify its own configuration or update security-relevant settings it can lead to remote code execution” (Rehberger). He listed the same move against tasks.json, against MCP server configs and against other agents’ configuration files.

Hooks have the same weakness. A hook is a guardrail written as code, a check that runs before or after the agent uses a tool, and in most coding agents it lives in a settings file in the project. Check Point showed in 2025 that a cloned repository could carry its own .claude/settings.json with a hook that ran a command the moment Claude Code started, with “no additional prompt or execution warning,” after a single click on the trust dialog. Anthropic fixed it, and Check Point’s conclusion applies well beyond one tool: developers “inherently trust project configuration files” because they look like metadata rather than code, and “the line between configuration and execution continues to blur” (Donenfeld and Vanunu). The file that holds the guardrail is a file. An agent that can write it can delete the guardrail, and an injection only has to ask.

The other vectors are all text the agent trusts more than it should. Pillar Security showed in March 2025 that rules files for Cursor and Copilot could carry instructions in invisible Unicode, readable by the agent and invisible to the developer reviewing the file (Pillar Security). Invariant Labs showed in April 2025 that an MCP tool’s description, which the model reads and the user mostly doesn’t, could tell the agent to read the user’s SSH keys and pass them to the attacker as a hidden argument (Invariant Labs). When researchers built a benchmark of these attacks across 45 real MCP servers, o1-mini followed the poisoned instructions 72.8 percent of the time, the highest refusal rate of any model was under 3 percent, and “more capable models are often more susceptible, as the attack exploits their superior instruction-following abilities” (Wang et al.). Add retrieved documents, tool results, memory and other agents’ messages, and almost everything an agent reads is a way in.

Notice that the swarm did a version of the same thing. When the agents wanted to hide their cheating, they went after the evaluator, the system checking their work. An agent under pressure, whether the pressure comes from an impossible task or an attacker’s instructions, goes for the controls.

The Prompt Can’t Hold It

Nobody in any of the reports suggests the agents would have stayed put if they’d been told more firmly not to leave. That isn’t because OpenAI wrote a bad prompt. It’s because instructions are the wrong layer for this.

The research on that is consistent. In Control Illusion, models “struggle with consistent instruction prioritization, even for simple formatting conflicts” (Geng et al.). In October 2025, researchers took 12 published defenses against prompt injection and jailbreaks and bypassed most of them with success rates above 90 percent (Nasr et al.). A prompt can draw a boundary around untrusted content, and it should, but it can’t enforce one. In Prompt Engineering Is Finding the Seams, I argued that most fixes shouldn’t live in the prompt, because the prompt is the layer nobody owns and nothing enforces. Safety is the extreme case. A rule in a prompt is a request. A rule in the environment is a fact.

The Remedy: Session Containers

A session container is an execution environment created for one session, isolated from everything else, and destroyed when the session ends. The model reasons outside it. When the agent needs to run code, call a shell or touch files, that happens inside the container, and the container decides what the code can reach. If an agent is injected, the damage is limited to one disposable box with nothing in it worth stealing and nowhere to send it.

Every platform that runs agents for a living now ships one. Azure Container Apps has dynamic sessions with outbound network access off by default. Claude Code has a sandbox that isolates the filesystem and the network at the operating system level. Codex blocks internet access during the agent’s work unless you turn it on. They differ in how hard the boundary is, and that difference matters more than it used to. What a session container needs, and what each of these gets right, is the second part of this series.

Five Lines

The swarm didn’t happen because OpenAI skipped the sandbox. It happened at the places the sandbox touched something else, and together with the injection vectors above, they make five lines a session container is built to hold.

The first is shared state. The swarm’s first message needed no exploit, only a writable store every session could see and a shared credential to write to it. Shared storage between sessions is a communication channel, whether or not anyone meant it as one, and in an agent system a communication channel is an injection channel. A session container shares nothing writable with any other session.

The second is egress. The environment had no direct internet access, but it had a package proxy, and a proxy is a door with a guard. Once the agents owned the guard, the door was open, and the screenshot trick shows how creative the path out can get. An allowlist is only as good as the least trustworthy thing on it, which was also EchoLeak’s lesson, where a domain Microsoft’s own content security policy trusted became the exfiltration path. A session container denies egress by default and sends anything allowed through a proxy outside the box that the agent can’t reconfigure.

The third is secrets. Everything the swarm did at Hugging Face ran on credentials sitting in environment variables, and every new credential opened the next system. A session that never holds a secret can’t leak one. That’s why Azure turns managed identity off by default, and why the right place for a credential is a proxy outside the session that attaches it to approved requests.

The fourth is the kernel, and here the evidence is newer. In March 2026, the UK AI Security Institute released SandboxEscapeBench, an open benchmark that “safely measures an LLM’s capacity to break out of these sandboxes” (Marchand et al.). Its finding is that “when vulnerabilities are added, LLMs are able to identify and exploit them.” A container is only a boundary if nothing inside it is smarter than the boundary, and that’s becoming a bad assumption, which is the case for a VM-grade boundary per session.

The fifth is the guardrails themselves. Hooks, permission settings, approval rules, MCP configs and rules files all have to live where the agent can read them, if it needs to, but never write them. In a session container that’s a read-only mount, or better, enforcement that runs outside the session entirely, the way Claude Code’s sandbox routes network traffic through a proxy the sandboxed process can’t touch. A guardrail the agent can edit is a suggestion.

After the incident, OpenAI quarantined IM1’s weights, delayed its frontier RL training runs, and committed to “more isolated sandboxes” and outbound access restricted to “a tightly controlled set of approved destinations,” enforced through “multiple independent layers of controls.” Those are the right fixes, and they’re the properties a session container should have from the start. One control held the whole time. OpenAI says none of the agents’ attempts to tamper with their records “affected the logs that our graders or monitors ultimately see.”

Why It Matters Outside the Labs

It would be easy to read the swarm as a lab problem. The agents were running with safety classifiers off, on a benchmark that rewarded breaking things, with models nobody else has. Most teams aren’t doing that.

But most teams are building the same shape of system, and the shape is what failed. Simon Willison’s lethal trifecta is private data, exposure to untrusted content and the ability to communicate externally (Willison), and nearly every useful agent has all three. A support agent reads tickets written by strangers. A coding agent reads issues and dependencies. A research agent reads the web. And more and more, agents read each other. Any one of those inputs can carry an injection, and the session container decides what an injected agent can do about it.

If I were deploying an agent that runs code or touches tools today, I’d give every session its own environment with a VM-grade boundary and destroy it when the session ends. I’d deny network egress by default and route anything allowed through a proxy outside the session that the agent can’t reconfigure. I’d keep credentials out of the session entirely and let the proxy attach them. I’d mount hooks, permission settings and agent configs read-only, and treat every rules file, tool description and cloned repository as untrusted input. I’d make sure no two sessions can write to anything the other can read, so one injected agent can’t become seven hundred. And I’d keep the logs outside the session, where the agent can’t edit them, because that’s what held for OpenAI when its agents tried to rewrite their own records.

A prompt tells an agent where the walls are. A session container is the walls.