SIGNALprophet format
Public AI leaderboards have a hidden slot machine. A model's standing on a benchmark is not a measurement of capability; it is capability multiplied by the model's willingness to follow the wrapper's expected output schema. A single model, on the same QA task, can post the low 40s under one prompt wrapper and the low 70s under another — same model, same task, opposite leaderboard rows.
Call it the wrapper lottery. The Format Sensitivity paper, a new arXiv preprint, puts a number on the gap. Across 140,000 OpenRouter generations on seven QA tasks, the Format Sensitivity Index varies by over 30x between models, and the variance is largely explained by parseability — the share of answers the model formats in a way the grader can actually read. A model that talks past the grader is not wrong; it is silent on the rubric.
The mechanism is portable. The harness picks a wrapper. The model emits a structured answer or it doesn't. The grader counts the unparseable ones as zero, and a model's standing moves by tens of points. The next benchmark runs the same loop with a different wrapper, and a different model wins.
The selection story matters first: benchmark suites are not random samples of "how a model performs" — they are samples of how a model performs inside a particular wrapper's grammar, a long way from deployment, where the wrapper is whatever the application supplies. The causal story is second: more compliant models look better, but compliance is a property of training recipes as much as capability.
The stakes are who claims leadership. Foundation labs, regulators drafting capability evaluations, and procurement officers all read the public numbers. Every unannotated score is a wrapper bet.
Heuristic: ask what the prompt wrapper was before you trust the ranking.