One signal, two jobs: a new architecture skips the text-tokenizer and only wakes part of the model at a time

Frontier model training burns cash, and the field is hedging with two parallel bets: drop the tokenizer so the model sees raw bytes, and route computation sparsely so most parameters sit idle most of the time. Until now those bets ran as separate projects with separate mechanisms: one engineered around byte patches, the other around learned router gates.

The EntropyMoE paper fuses them with a single coordinate. Patch entropy, the same measure that decides how bytes get grouped, also decides which expert fires on a patch. One signal does both jobs, and the feature space that emerges, entropy plus length, gives experts a specialization inspectable on the page rather than buried in learned weights.

Compute expands where bytes carry surprise and contracts where they do not, with no separate learned gate. The EntropyMoE authors report the lowest held-out bits-per-byte against matched dense and sparse baselines, with downstream accuracy comparable rather than improved. The honest read: the gain is matching the prior frontier on a fairer axis, not clearing it.

The portable test: when the next tokenizer-free MoE paper arrives, ask which coordinate it routes on. A learned gate means the old bet in new clothes. An inspectable signal aligned with the segmentation rule means a step along the same mechanism EntropyMoE names.