SIGNAL
Every "Model X beats Model Y" headline is hiding a second variable, the code wrapped around the model. A benchmark score is a joint product of the system being tested and the harness doing the testing: how observations are fed in, how the model is allowed to think between moves, and how much of that thinking it is allowed to keep. Most leaderboard comparisons pretend the harness is a constant. It is not.
OpenAI's own ARC-AGI-3 post is the year's cleanest demonstration. A frontier model that helped close a long-standing conjecture and cleared Pokémon FireRed scored 7.8% on a public 2D-puzzle benchmark, then roughly tripled that number once two routine API settings were turned on. The model did not change. The two settings, retained reasoning and compaction, control how much intermediate thinking the model carries across actions. With them off, the model re-derived the game state from scratch every move, even though past moves sat in plain view. With them on, it learned.
The lesson isn't that ARC-AGI-3 is broken. Its intentionally generic harness is a defensible choice: it surfaces model shortcomings without letting clever tooling paper over them. The lesson is that a single score from one lab, with no harness disclosure, is a snapshot of a measurement, not a property of the model. Cross-lab leaderboard claims that don't report configuration are comparing setups, not just systems.
The ask is small and portable: report harness configuration next to the score. Until that becomes standard, the right reader-side filter is two words: "which harness?"
Reported by Sky for Type0, from How enabling two settings tripled our scores on the ARC-AGI-3 benchmark. Read the original: openai.com