AIBRIEF
Training an AI by reward grows a layered architecture where deep layers do the math reasoning; training by example spreads capability evenly across every layer.
When researchers trained an AI to solve math problems by rewarding correct answers, the model didn't just score better. It grew a layered internal architecture in which the deepest layers do the bulk of the reasoning. Train the same model by showing it worked examples instead, and capability spreads evenly across every layer, leaving no clear "thinking" tier to point to. That contrast is the central finding of Probing the Origins of Reasoning Performance, a new mechanistic-interpretability paper comparing reinforcement learning (RL) and supervised fine-tuning (SFT) on math-reasoning benchmarks.
Reinforcement learning rewards the model for correct answers; supervised fine-tuning shows it solved examples and asks it to imitate. Both methods lift benchmark scores, but the paper asks what changes inside.
The team trained linear probes, which are simple classifiers that read a model's hidden state at each layer, to predict whether an answer was correct. Probes on the RL models reached higher accuracy, the paper reports, suggesting the internal representation of "is this answer right?" is more cleanly organized. The HTML version walks through the layered plots.
A second experiment removed, or "ablated," each layer in turn. In RL models, deeper layers were progressively more critical; disabling them collapsed performance. In SFT models, importance was distributed roughly evenly across the stack.
The paper carries one honest caveat: token-count variability under repeated sampling is mixed across RL models. Some vary more than their SFT counterparts, others do not. The authors read this as evidence that token behavior tracks the whole training pipeline, not RL versus SFT as a clean binary. An author-hosted code supplement ships with the paper.