A 2.8M-parameter AI beat frontier models on a controlled reasoning test

A 2.8-million-parameter transformer matched the Bayesian optimum on a controlled model-selection task, and frontier-scale language models got the same probe wrong by roughly 55 times. The result, from a preprint posted to arXiv, is a controlled test of when "just train a bigger model" stops working, and what the bottleneck actually is.

The paper introduces what its authors call "Bayesian wind tunnels": synthetic environments with closed-form ground truth, where the correct answer is known in advance and can be checked bit-for-bit against a model's output. The test bed contrasts two deliberately non-nested hypothesis classes, fixed-point-free involutions (functions where f(f(x)) = x) and 3-cycles, so neither class is a subset of the other. That removes the usual shortcut where a more expressive class simply contains a simpler one.

On the integer-token version of the task, the 2.8-million-parameter transformer reached 0.01-bit entropy agreement with the Bayesian optimum, within three seeds, on the involution class-posterior problem. The same architecture also handled 3-cycles, achieving class-posterior mean absolute error under 0.001. The authors describe this as genuine Bayesian model selection, not a nesting-driven approximation.

They rewrote the test using opaque symbols whose meanings change every episode. The discriminative statistic now required modular addition (rotations in a finite field) or multiplication (f(x) = cx mod p), but the tokens looked like arbitrary labels. The small model failed. The authors then scaled the same architecture 112x, from 2.8 million to 316 million parameters, and re-ran the test. The gap did not close.

A separate "header subtask" diagnostic pinned the failure to the composition of header inversion with arithmetic, not to header parsing itself. The model could read the structure of the problem; it could not execute the operation. The paper's authors call this a "perceptual access condition": the architecture needs integer-like tokens to compile the relevant circuit. The intuition is closer to long division over a string of digits than to arithmetic in the abstract. The model can name the rule; it cannot reach the operands once they are hidden behind opaque labels, and the compile step is what scaling does not rescue.

A stationarity control makes the claim falsifiable. When the opaque tokens carry a fixed relabeling (stable semantics, just not integer-shaped), the same architecture succeeds at 0.009-bit mean absolute error. Stable semantics are enough; integer identity is what unlocks arithmetic. If the bottleneck were parameter count, this control would not work.

Frontier-scale language models probed on the same tasks show qualitative Bayesian behavior: their answers move in the right direction under evidence. The paper reports a roughly 55x calibration gap on the probe, meaning their stated confidence does not match how often they are right. The authors frame this as directional, because the probe reads the model's beliefs through a lossy lens, not as a precise benchmark. The largest available models misstate their own confidence by more than an order of magnitude on this probe.

Scaling helps when the architecture can already see the relevant structure in its tokens. Scaling does not rescue representational blind spots. The right intervention, on this evidence, is token shape, whether the model can read arithmetic structure in the input, not parameter count.

The paper is a preprint, not peer-reviewed, and the strong scaling-robustness result is confined to the involution/3-cycle distribution. The authors are explicit that it does not imply real-world generalization. The frontier-model result is also confined to a single probe. The paper supplies a controlled, falsifiable experiment at a moment when the "scaling is hitting limits" debate is mostly vibes. The next test is whether other reasoning benchmarks break the same way when their tokens are hidden behind opaque labels.