Fuzzing, the decades old bug hunting technique ported from software security to AI red teaming, has been reported to break safety tuned ChatGPT and Llama 2 7B guardrails on more than 90% of attempts. The same loop is becoming routine evaluation work for defenders.
A red-teaming pipeline borrowed from classical software security is now beating the safety filters on the most heavily tuned commercial language models, with no human in the loop. GPTFuzz, a black-box fuzzing framework introduced in 2023, reports attack-success rates above 90% against ChatGPT and Llama-2-7B and above 85% against Vicuna-7B by automating the same kind of coverage-guided mutation loop that has been breaking software bugs for two decades (GPTFUZZER paper, Yu et al., arXiv:2309.10253).
The point is not the number. The point is the loop.
Fuzzing is one of the oldest ideas in software security. American Fuzzy Lop (AFL), released in 2013 and descended from earlier coverage-guided fuzzers, bombards a target program with mutated inputs and watches which inputs reach new code paths; promising seeds get retained and mutated further. It is a Darwinian search: throw variations at a system, keep the ones that change something, repeat. Security labs have used the technique for twenty years to find crashes in browsers, file parsers, and network stacks. Now the same idea has been ported to language-model red-teaming, and the port is more disruptive than the original.
GPTFuzz starts with a small library of seed jailbreak prompts, human-written examples that already push a model's safety filters. It then uses a separate language model to mutate those seeds into thousands of candidate prompts. Each candidate is run against the target model, and an automated check scores whether the response violates the safety policy. Candidates that succeed are kept, added to the seed bank, and mutated again in the next round. The loop runs without a human in the chair (Yu et al., arXiv:2309.10253 HTML). The numbers above are what fall out of that engine after a modest number of iterations.
The empirical result is uncomfortable. The reported >90% attack success rate was not against an open-source toy model. It was against ChatGPT (GPT-3.5 and GPT-4 generations) and Meta's Llama-2-7B-Chat, both of which had been hardened with reinforcement learning from human feedback and adversarial training. The method worked because the mutation space is huge and the loop is patient: humans writing prompts by hand explore a few corners, but a fuzzing-style pipeline keeps grinding through variations and keeps the ones that slip past filters (Yu et al., arXiv:2309.10253).
The technique is not a one-off. An independent peer-reviewed paper at USENIX Security 2024, "LLM-Fuzzer: Scaling Assessment of Large Language Model Jailbreaks," from the same author family, confirmed that automated mutation-and-selection red-teaming scales beyond small seed sets and works across multiple target models (Yu et al., USENIX Security 2024). The category now has a name, automated jailbreak fuzzing, and at least one open-source reimplementation, sherdencooper/GPTFuzz, that anyone can clone and run, though readers should treat that repository as a community reproduction rather than the original authors' canonical artifact.
The second-order story is what makes this more than a vulnerability disclosure. For two decades, fuzzing moved software security from manual bug hunting to a continuous engineering discipline with shared harnesses, public corpora, and reproducible benchmarks. The same shift is starting in LLM safety. A vendor that ships a new model today is expected to run an internal red-team sweep; the open question is whether that sweep is a checklist of known prompts or a fuzzing pipeline that continuously generates new ones. Coverage-guided adversarial testing is becoming routine work for model safety teams, not because it is exotic, but because it is the cheapest way to find the holes a static guardrail list will miss (SemiEngineering analysis).
That framing also gives defenders a constructive read. The loop that breaks ChatGPT can be turned around: any team deploying an LLM can run the same coverage-guided mutation harness against their own model, score their own safety policy, and keep the seeds that fail. The same pipeline that surfaces attacks also produces a regression corpus for the next training run. The cost curve for both offense and defense has compressed at once.
The caveats matter. The headline numbers come from a 2023 arXiv preprint (arXiv:2309.10253), so the empirical claims rest on the paper's reported tables rather than peer-reviewed publication of that specific artifact. Model providers have shipped guardrail updates since then, and the published >90% figure is a snapshot of those models at the time, not a permanent score against current ChatGPT or Llama releases. Independent peer-reviewed work from the same author family at USENIX Security 2024 corroborates the broader automated mutation-and-selection thesis but is a separate artifact (Yu et al., USENIX Security 2024). The sherdencooper/GPTFuzz repository is a community reimplementation rather than the original authors' canonical code; readers replicating the loop should verify against the paper's reference implementation. And the broader coverage of this beat, including the SemiEngineering explainer that surfaced this story, sits inside a vendor framing context where protocol-fuzzing expertise is part of the publication's commercial beat (SemiEngineering).
What to watch next: whether major model vendors publish fuzzing-style red-team results in their model cards the way cloud providers publish fuzzing corpora for their parsers, and whether the open-source jailbreak-fuzzing ecosystem settles on a shared benchmark so an attack-success rate on Vicuna-7B in 2026 means the same thing it did in 2023. The technique that broke software bugs first is now the technique being asked to break AI guardrails, and the answer so far is that it does.