The harness ate the model

The engineering effort behind a benchmark score has migrated out of the model and into the harness that wraps it. A score on ARC-AGI-3 now reflects the process around the model: how observations are turned into a working internal state, how predictions are tested against history, and how plans are revised when they fail. Weights stay frozen; the work happens in the code that runs them.

When a small open-source project called Schema claims 99% on the ARC-AGI-3 public set using Claude Opus 4.8, Fable 5, and GPT-5.6 Sol, the headline-worthy object is not any one model. It is the per-game fallback rule: Opus 4.8 and Sol xhigh run first, sub-80 games get rerun with Fable 5 or Sol max, and the higher per-game score is kept. That is ensemble by selection, not by learning, and it does not change the underlying model weights.

The credible alternative is that better prompting alone is doing the work, and Schema is exposing capability the models already had. The control that would settle that, a no-fallback run of Opus 4.8 and GPT-5.6 Sol in isolation, is missing from the Reddit post. Until it ships, the score is a claim, and ARC Prize has not verified it; the organization's president has only said "Looks cool, need to dig into it."

The reader's new default question: when a number like this lands, ask which lever moved. The model, the data, or the process. If replication holds, the next round of "X% on benchmark Y" headlines will be measuring wrappers, and the people who know the difference will read them differently.