When Tools Hunt for the Code Change That Introduced a Bug, They Get It Wrong Two-Thirds of the Time

Automated tools that try to identify the exact code change that first introduced a known security flaw correctly pin it down only about a third of the time on a new, harder test set, because the bug-introducing commits they are searching for are, on average, more than six times larger than the fixes that later patched them.

The number comes from VICBench, a new benchmark of 100 verified vulnerability-inducing commits (VICs) tied to 100 CVEs across 88 open-source projects in Python, Java, and C++. State-of-the-art VIC-identification algorithms V-SZZ and LLM4SZZ scored between 33.3% and 40.1% F1 on the test, the authors report, well below the levels those tools have hit on older, Java-heavy benchmarks.

A VIC is the code change that first introduced a known security bug. Pinning it down tells defenders every version of a software product that should be treated as at risk, which sets the floor for how fast a patch can reach users. F1 is a standard accuracy measure that balances false positives and missed hits; a score of 33% to 40% means the tools are right far less often than they are wrong on this test.

The benchmark's central finding is structural, not just a low score. Real-world vulnerability fixes in VICBench average 38.6 lines. The corresponding VICs average 252.5 lines, which the authors call a significantly larger change footprint than prior benchmarks captured. A six-and-a-half-times size ratio means the bug-introducing commit is rarely a small, surgical change that an algorithm can spot by diffing forward from a known fix. It is more often a sprawling, multi-purpose commit that mixes the flaw with unrelated work: refactors, formatting, feature scaffolding, dependency bumps. The signal of the original bug sits inside that larger mix.

SZZ-family algorithms, which backtrack from a known fix to the commit that introduced the bug, struggle for a structural reason: they look for the smallest plausible change. Bug-introducing commits, in the wild, are rarely that.

The benchmark was constructed by hand. Two human annotators walked 100 CVEs back to the specific commit that introduced each flaw, with an agentic workflow assisting cross-language coverage. The result spans 48 CWE categories, the common taxonomy used to classify software weaknesses, and includes 88 projects across three of the most-used open-source languages. Coverage that broad is unusual: most prior VIC benchmarks concentrated on a single language, which made their bug-introducing commits look smaller than they actually are.

Operationally, the low F1 means anyone trying to answer "which versions of this library are affected?" still leans on human auditors to confirm the VIC, not on the automated backtracking tools. Until algorithms can reason about intent across hundreds of lines of mixed-purpose changes, patch velocity stays bounded by the speed of that human review. The asymmetric burden shows up directly in CVE response: a maintainer who cannot confidently identify the VIC has to either patch every release between the fix and the bug's introduction, or wait for an auditor to narrow the window.

The benchmark is a research contribution, not a vendor- or regulator-endorsed standard. The arXiv preprint has not been peer-reviewed, and the headline numbers are author-reported until independent teams exercise the test set. The strongest counterargument is that VICBench may simply be harder than typical real-world distributions, which would make the gap a benchmark effect rather than a true performance ceiling. In either case, the operational consequence holds: the tools defenders rely on to scope affected versions are correct roughly a third of the time on a curated, multi-language test, and the rest of the work still falls to people.