AI Is Re-Auditing a Century of Science, and Finding Errors

A theoretical chemist at a Zhejiang lab asked an AI to predict molecular boiling points. The model flagged the 75-year-old reference database it was drawing on as wrong. A follow-up audit also flagged a roughly century-old authoritative boiling-point measurement as wrong. Both had been absorbed into mainstream chemistry without anyone re-checking the underlying number, according to a recent write-up of the case.

That single correction captures the shift underway across the sciences. The same cost collapse that made large language models possible is now letting researchers re-run other people's experiments and re-check other people's data, so a century of published literature is, for the first time, systematically re-auditable. The audits are surfacing a structural strain, not a single scandal. Peer review was not built for the volume or the citation chains that AI-era science has produced.

The most concrete snapshot comes from this year's ICML 2026 oral reproduction audit, which tried to re-run the headline experiments of 168 accepted papers at ICML, one of the three top annual AI research conferences alongside NeurIPS and ICLR. Of those, 92 had at least five claims that could be checked against released code and data. Only 34 of those 92 papers reproduced more than 40 percent of their conclusions. Only 8 reproduced more than 80 percent. The failure modes were mundane and instructive: missing key files, broken or legacy dependencies, runtime mismatches with the stated environment, and four papers whose underlying models had been decommissioned and were therefore permanently unreproducible.

A separate thread is running in parallel. A team behind the GPT-5-based Paper Correctness Checker built a tool that scans published papers for objective errors such as wrong formulas, derivation gaps, faulty proof assumptions, and missing methods. Across the papers it scanned, the checker found an average of 4.7 objective errors per paper. 99.2 percent of scanned papers were flagged with at least one issue. Math and formula errors made up 54.0 percent of the catch. The same audit found that 30.8 percent of NeurIPS papers and 23.8 percent of ICLR papers contain at least one substantive error that changes how the result should be read. One concrete case from the underlying paper: a study marketed as training 0.77 percent of base-model parameters had actually released a checkpoint trained on 6.31 percent, a roughly eightfold gap between the headline and the artifact.

The third thread is the contest. Hugging Face and AlphaXiv are running the Agent Reproduction Challenge, inviting researchers to use AI coding agents to reproduce ICML 2026 papers, with an open campaign repository tracking the leaderboard. The premise is that reproduction is no longer a human-only bottleneck: an agent can install the environment, run the script, and report whether the numbers match.

The trend, on the evidence available so far, is getting worse rather than better. Average objective errors per NeurIPS paper rose from 3.8 in 2021 to 5.9 in 2025, a 55.3 percent increase. ICLR annual submissions grew from 1,013 in 2018 to 19,619 in 2026. The mechanism for silent propagation is straightforward: a single early error, once cited downstream, becomes citation-chain consensus because manual re-verification is high-cost and low-reward.

The ceiling of the new audit sits at 83.2 percent detection precision, and the checker still misses roughly 40 percent of real errors per scan. That is enough to catch a wide class of mechanical mistakes. It is not enough to settle disputes, so human review of the AI's output remains required. The Pios chemistry case worked because the corrected measurements were checked against fresh calculations; the AI's flag was the start of the correction, not the verdict.

The live test is the Agent Reproduction Challenge. The first wave of results will land in the next few months, and they will determine whether the 99.2 percent figure becomes a permanent feature of the literature or a transient one as the verification layer scales.