QCNEWS
IBM, Hugging Face and the Technical University of Munich pooled 22,000 results to track the gap between AI grading harnesses, the code that runs and scores a test.
Two teams can run what looks like the same exam on the same AI model and post scores that differ by up to 20 percentage points. The gap usually isn't in the model. It's in the "harness," the code that administers a test, scores the answers, and reports the result. A new community-led database called EveryEvalEver is the field's first attempt to put that variance on the public record rather than let it hide inside leaderboards.
The database, hosted on Hugging Face and built by researchers at IBM, Hugging Face, the Technical University of Munich, and other community contributors, now lists more than 22,000 model results across 2,200 benchmarks, drawn from 31 different evaluation formats that have been translated into one common schema. The IBM Research blog post introducing the project frames it as community work to raise the scientific bar on AI evaluation, not as a vendor fix.
The 20-point swing is concrete. Researchers at Hugging Face have separately observed conflicting scores for the same LLaMA model on MMLU, a standard multiple-choice knowledge test used to compare large language models. Two harness implementations of "the same evaluation" can disagree on the answer key, the prompt format, or the way multiple correct answers are scored, and that disagreement can be the entire spread.
"There are so many, and they're so different from each other, that almost every decision is based on gut feeling, because we don't have the data," IBM researcher Leshem Choshen said in the same blog post, describing how teams pick which benchmark to trust.
The fragmentation runs deeper than the harness. AI benchmarks are not standardized tests in the way a college entrance exam is. They are usually static question sets, published online, with no live proctoring, and they are routinely used to train the same models they are supposed to evaluate. That overlap between training data and test data is a separate, older source of score inflation. EveryEvalEver's job is the narrower one: it makes the harness-level differences auditable, the same model on the same test, run through different grading code, with the settings, prompt format, and scoring rules visible side by side.
The numbers, the 22,000 results, the 2,200 benchmarks, the 20-point variance figure, the LLaMA-on-MMLU case, and the "gut feeling" quote, come from an IBM Research blog post authored from inside the team that helped lead the project. IBM, Hugging Face, and the Technical University of Munich are not external auditors of the AI benchmarking problem; they are contributors to the proposed fix. Independent confirmation of the specific variance figures from one of the other named leads would tighten the case, and the numbers here are reported as IBM characterizes them.
The database does not standardize the harnesses, retire the contaminated benchmarks, or settle the question of which leaderboard a model buyer should trust. It collects the existing scores, the harness that produced each one, and the settings, and makes the comparison possible. Whether the variance gets smaller is a separate question for the community that maintains the harnesses, not for the project that counts them.
The watch item, for anyone using a leaderboard to make a model-buying decision now, is whether the harness that produced a given score is named in the result. If it isn't, the 20-point spread is the default assumption.