Before You Train an LLM, This Data Test Forecasts Its Weight Growth

Before you spend GPU hours training an LLM, a single entropy statistic computed from the training corpus alone can forecast how the model's learned parameters will grow during training. The result is in arXiv preprint 2608.23573, which recovers held-out within-family weight growth to within 5.7% relative error across 23 training runs spanning an order of magnitude in learning rate. The paper flags, on its first page, the cross-corpus case where the predictor over-predicts.

The statistic is a training-free bigram conditional entropy: the entropy of the next token given the previous one, computed by counting two-token pairs in the corpus. Call it D. The same statistic on a randomly shuffled version of the corpus, call it H_r, sets a baseline. The training-time variable the paper tracks is the scale parameter of a Weibull distribution, a two-parameter shape commonly used for failure-time and size data, that fits the magnitudes of the transformer's learned weights. Across layers and across training runs, that distribution's shape is stable at about k ≈ 1.2. What changes during training is the scale, λ, and almost all of the training-induced movement lives there.

Plotted against (H_r − D)^0.59, the corpus's predictability above a shuffled baseline raised to a fixed exponent, the per-run growth falls on a unit-slope line with R² = 0.941. The 0.59 exponent is not fitted; it is inherited from an independently measured data-side saturation relation. Per-learning-rate fits are weaker at R² ≈ 0.82, which is the more useful caveat: the unification is what the data does, not what the per-run fit can be coaxed into. The learning-rate dependence survives as two coefficients, C₀(η) and C₁(η), on a law of the form λ² − λ₀² = C₀(η) + C₁(η)·(H_r − D)^0.59, where λ₀ is the scale at the start of training and the squared-scale difference tracks how far weights have moved (HTML, code).

The reader payoff is concrete. A builder can compute D from the training corpus before any model is trained, pick a learning rate, and estimate one slice of how the model's weights will grow as training proceeds. The workflow is small: tokenize the corpus, count two-token pairs, compute the conditional entropy, shuffle the same corpus under a matched token budget to get H_r, and plug the difference into (H_r − D)^0.59. The 5.7% relative error on held-out within-family growth is the diagnostic's ceiling on the configurations the authors tested. The readout also holds at per-layer resolution and across two tested architectures; only the coefficients change.

When the same predictor is taken across corpora, it over-predicts on code, and the authors name the missing axis: data redundancy, R. The broader framework they gesture toward, written Φ(D,R,A,H), would add redundancy to a data-to-weight mapping the current law only partially spans. That is also the load-bearing direction for anyone who wants the diagnostic to survive a real pre-training mix rather than a matched corpus. Only two architectures appear in the readout, and the forward-predictor claim depends on D being computed the same way downstream; the reproducibility hinge is the matched-budget shuffle baseline H_r.

What the law doesn't say is also part of the result. The Weibull scale λ captures the size of learned parameters, not loss, not generalization, and not downstream task accuracy. The work is an arXiv preprint, not a peer-reviewed paper, and the experiments are about a single family of transformer training. Treat the 5.7% as a ceiling on a specific diagnostic, and treat the code-over-prediction as the named failure mode a follow-up would have to address. Whether redundancy alone closes the gap is the watch item.