You can't verify what an AI trained on. So change the test.

Frontier AI leaderboard scores look like measurements. They are audit receipts, and the receipts come from the party whose score depends on them. A structural argument, laid out in a GitHub Pages essay from Holdout Labs, says today's benchmark numbers cannot be independently checked because the only party who can verify "we did not train on the test" is the lab that ran the test. The essay proposes a constructive replacement: a four-part evaluation-side rule that takes the verification floor out of the lab's hands.

According to the essay, OpenAI retired SWE-bench Verified in February, the benchmark most coding-model announcements had been built on, and recommended that other labs stop too. The stated reason: every frontier model OpenAI tested could reproduce the human-written reference fix, or verbatim details of the problem statement, for some tasks. The models had encountered material that gave away the answers.

SWE-bench Verified is a curated set of real GitHub issues, paired with the human-written patch that resolved them. A model scores by producing a patch the test suite accepts. "Reproducing the human fix" means producing the same code change the original engineer wrote.

The deeper problem is what that retirement reveals. A benchmark score is portable only if a third party can rerun it. The training side, meaning what a lab put into the model before testing, is opaque. The best a lab can do is publish a decontamination report: a claim that the test set, plus paraphrase variants, was not in the training corpus. The claim is checked against the corpus nobody else can inspect, using a matcher that misses paraphrase and synthetic derivatives, by the party whose score depends on the result.

PSI (Private Set Intersection) lets a lab prove that a specific test item is not in its corpus, but it does not extend to a full set check. "Proof-of-training" proposals have been shown to be spoofable in published work. Even the most aggressive cryptographic framing, a zero-knowledge (ZK) proof that the model was trained on a specific dataset, would still have to bind the model, the inputs, the scoring, and the evaluation together to be meaningful. A proof that an inference was run correctly (proof-of-inference) is not the same as a proof that a benchmark result is real.

Holdout Labs, a small vendor building a verified-evaluation product, calls the decontamination report "an audit receipt, not a portable proof." The point applies to every frontier-lab leaderboard score a reader has seen: the verification floor has to move from training-side self-report to evaluation-side rules.

The four-part rule, in plain English. First, the submission never receives labels: the model returns an answer, the evaluator holds the answer key. Second, no network at evaluation: the model is air-gapped, so it cannot phone home to a copy of the test set. Third, the evaluator rebuilds the system from a named commit, a specific tagged version of the code on GitHub, so anyone can reproduce the exact configuration. Fourth, where the problem allows it, the test data is forward-dated: the questions are written after the model is frozen, so there is no way the model could have seen them during training.

The four limits the author concedes. The rule does not, by itself, prove benchmark validity, meaning that the test measures what it claims to measure. It does not, by itself, prevent adaptive overfitting through repeated submissions, because there is no per-solver submission budget in the proposal. It does not, by itself, address funder-side leakage, where a funder with early access to the test set influences training. It does not, by itself, make a benchmark third-party re-runnable without the data: the test set still has to be held under controlled conditions for the rule to mean anything.

The author has implemented the rule for small tabular models, the kind used in healthcare and credit scoring, and explicitly separates what is built from what is design. The proposal is not a finished standard; it is a working draft that the author is asking to be critiqued, especially by anyone running a persistent private leaderboard.

A vendor's preferred standard is still a vendor's standard, and the reader should hold the claim at arm's length: Holdout Labs sells the kind of evaluation the rule would require. The argument stands on its own, that a self-report over an opaque corpus, by the party whose score depends on the result, is not independently checkable. The constructive rule is the response.

The next milestone the reader can watch is whether any frontier lab, or any independent evaluator, adopts even one of the four conditions (no labels at submission, no network at eval, named commit, forward-dated data) for a high-stakes benchmark. The essay is published on GitHub Pages and the author's critique invitation is open.