1Password's 74% AI-patch failure rate is the wrong number

1Password's research arm ran frontier AI systems against six real software bugs and graded the patches. Twenty-six percent landed clean. The other 74% looked like fixes until a reviewer read them.

1Password's Off-By-1-Labs named the failure shape: FLAWED, Fix-Like Artifacts With Embedded Defects. The most common variant was the patch that removed the vulnerable code path and the feature that path served. Static analysis passes, the build passes, the CVE checker stops flagging, and no regression test exists. A human reviewer catches it. A CI pipeline that only checks "does it compile and does the existing test suite still pass" does not.

The six bugs were chosen so the models would not have seen them in training. The set: a Linux kernel privilege-escalation flaw (a Copy Fail in copy_from_user), an ActiveMQ remote code execution bug, a Chrome File System Access API use-after-free on macOS, an unauthenticated remote code execution flaw in the EXIM mail server, a Spring AI SpEL (Spring expression language) injection, and a command-injection bug in Google's Gemini CLI. On the hardest patches in the set, 1Password's own writeup puts the FLAWED rate at 53.9%. The 74% number in ZDNet's coverage is the inverse of the 26% that landed clean across the full test. Different metric, same paper.

Two model families were in scope: Anthropic's Claude and an LLM based on OpenAI's Codex coding agent. The team is explicit that the study is not a head-to-head ranking. The team's pre-study guess was about 67% patch success; actual results were, in their words, "significantly lower and more uneven than we hypothesized." The harness itself is the durable artifact, released on GitHub as FLAWED: it checks out an open-source project at the known-vulnerable commit, hands an AI agent the bug description, generates a patch, validates it, and runs an AST check. Only the clone stage has network access; the agent never sees the real upstream fix.

The timing is what the wire left out. Anthropic's Project Glasswing and OpenAI's June 2025 agentic-coding push are both moving vulnerability discovery on the same exponential curve. If finding vulnerabilities has scaled and fixing them has not, the remediation backlog grows. The honest version of the takeaway is a workflow split: hand the model the parts of patching that look like search (surfacing candidates, drafting a starting patch, ranking options), and keep the parts that look like judgment (review, merge, ownership) on a human.

This is one vendor's research arm, on six hand-picked CVEs, using a methodology the vendor itself published. The 26% that landed clean is a small absolute number, and the FLAWED taxonomy is a single team's read of the failure modes. The headline "AI patches are broken" survives the study. The headline "AI patches are useless" does not. The paper's own number is the right summary: roughly one in four AI-generated security patches is safe to merge without a human rewriting it. The deployment decision is which part of the workflow to put the human in.