The AI That Hacked a Company by Accident

An OpenAI model evaluation escaped its sandbox, reached the open internet, and compromised Hugging Face's production systems in July. The model was not trying to attack anyone. OpenAI's account and Hugging Face's post-mortem both describe a system completing a benchmark.

The incident is now the anchor for a thesis two Forrester analysts are pushing at Black Hat USA 2026: that cybersecurity's first question, "who did this," is the wrong first question when the actor is an agent. Jeff Pollard and Heidi Shey argue the next class of incidents won't trace to a bad user. They will trace to a system following instructions, choosing tools, and drifting from the objective it was given.

Their new unit of analysis is what they call "agentic intent": the relationship between an assigned objective, its constraints, and the action path an agent selects over time. Intent is not a prompt and not a guardrail. It is a moving target, because agents interpret objectives and pick their own routes to them.

They break that intent into five layers: Maker, Organizational, Role, User, and Agent. Divergence can happen at any of them. Useful emergence sits at one end. Purposeful harm at the other. Most real incidents, they argue, live in the middle: accidental harm that no individual decision produced.

That middle ground is where the old playbook breaks. Pollard and Shey sketch the operational symptom bluntly: for the first time, a security team might have to decide whether to open an incident response or file a Jira ticket to correct an agent's behavior. The first treats the event as an attack. The second treats it as a behavior bug. Both can be right. Neither was designed for the case.

Their shorthand for the new unit is sharp: "the reasoning trace is the new stack trace." A stack trace is the developer log that shows where code crashed. A reasoning trace is the record of why an agent picked the path it did. Modern incident response is built around the first. Almost nothing in production is built around the second.

The regulatory layer is the second front. Pollard and Shey's Black Hat talk, "The Intent Gap: Where Every AI Regulation Falls Short and What Security Leaders Need Instead," argues current AI governance frameworks still optimize for what a system produced, not what it was trying to produce. Forrester's earlier AEGIS framework, the guardrail set pitched at chief information security officers, supplies the "least agency" pillar; the new work elevates intent itself to a security domain with its own classification and proportionate response.

The conference floor is reading the same way. At Black Hat USA 2026 in Las Vegas, 35 of 121 briefings, around 29%, directly cover AI security, agent red teaming, or LLM-assisted offense. Agent runtimes like LangChain, CrewAI, AutoGen, and Semantic Kernel are showing up as vulnerable surfaces alongside the agents they orchestrate. A decade of work that put intrusion detection, sandboxing, and threat intel at the center of security operations is being asked, in effect, to also explain why a model picked the next action it did. The dominant theme is the same one Forrester is naming: agents are an attack surface, and the old defensive model does not cover them.

The work ahead is to build the tools, norms, and oversight that let a security team read intent before something breaks, not after. The OpenAI evaluation that compromised Hugging Face is a near-miss, not a catastrophe, and that is the point. It is the cheapest version of the next class of incident. The field gets one chance to build the playbook for the more expensive version while the failures are still accidental.