AINEWS
The textbook says repeatedly checking the same test set should corrupt it, so why do ML leaderboards still track real progress?
By every statistics rule, the way the field chases the same ML leaderboards should have broken them by now. The textbook is explicit: a model that gets checked against the same test set, tweaked, and checked again will eventually memorize the test rather than learn the task. Reuse a held-out set enough times and you have folded it back into training; the "genuinely unseen" condition that gives the score its meaning is gone, and the model that wins the leaderboard is the one that has best learned to game it.
ML research is, by that logic, a textbook disaster in slow motion. A small set of long-lived benchmarks (ImageNet for vision, GLUE and MMLU for language, and a growing catalog of newer ones) gets checked by thousands of teams over many years in what is effectively a distributed, multi-year hill-climb. If the textbook is right, the leaderboards should now be saturated with models that look great on the benchmark and mediocre everywhere else.
They are not, and that gap is the puzzle an Amazon Science post openly concedes. The piece poses a research question, not a verdict, and that posture is deliberate: nobody is claiming benchmarks are fine. The claim is narrower. The lack of obvious collapse is itself a finding, and the field is still arguing about what produces it.
The leading candidate explanation, and the one the post points to, is some form of compression pressure: a bias toward simpler hypotheses that generalize, even when researchers are explicitly optimizing for benchmark scores. Borrowing the post's own language, this is Occam's razor made mechanical. The iterative training-and-pruning loop appears to reward models that capture the underlying pattern rather than memorize the test. It is the most interesting answer on the table, but it is not a closed case. If gains on standard benchmarks suddenly stopped transferring to fresh, held-out evaluations, the compression story would be in trouble.
The question is freshly pressing because the iteration loop is getting faster. A modern LLM-driven research system can run the train-check-tweak cycle in hours rather than weeks and parallelize it across thousands of configurations at once. Whatever mechanism protects benchmarks from collapse is being stressed in a way the field has not yet measured, and the post's own title, "research agents," points at exactly that sharper version of the puzzle. A mechanism that held up under human-paced research may behave differently under agent-paced research, and the field does not yet have the data to settle that.
A Hacker News thread on the post surfaced the same questions readers are asking. Most of the comments flagged the post's pacing and one graphic labeled "Occam's razor, formalized" as confusing, which is a fair critique of presentation. What stops the leaderboards from collapsing under their own weight is the substantive question, the one the post is trying to surface and the one the field is still trying to answer.
If a new generation of held-out, contamination-resistant evaluations starts to diverge sharply from the long-lived leaderboards, that is the signal to watch. Until then, the puzzle stands: a research culture that should, by its own rules, be deceiving itself, and a body of empirical evidence that says it mostly is not. The next time a model tops a benchmark by a clean margin, the honest question is not whether the score is real. It is what, exactly, is keeping it real.