Encoders Are a Size Problem, Not a Modality Problem

Every model needs a translator that turns raw audio and video into something it can read. Until last month, that was treated as a law of nature. The Gemma 4 technical report shows it was a budget line.

Below a certain scale, the encoder is a cheap bridge: a 305M audio component, a 150M vision component, compressed from 390MB to 87MB, and still posting a 17% transcription accuracy lift on the E2B. Above that scale, the encoder becomes dead weight, because the new attention stack (5:1 local-to-global, value tensors reused as keys, p-RoPE at p=0.25) carries enough memory to learn from raw modality tokens on its own.

That is why only the 12B ships without an encoder. The smaller E2B and E4B keep theirs and benefit; the 26B-A4B and 31B keep the 550M vision encoder and drop audio. The pattern is size, not modality. Sofia Reyes's read of the technical report in Cherry Creek News flags the number that ties it together: a 37.5% reduction in the global KV cache, the deployment-relevant payoff of the new attention stack.

The stakes for builders are practical. If the encoder is redundant at 12B and useful below it, then the compression path, the shipping footprint, and the inference budget all change. Coverage that flattens the result into "Google removed the encoder and the model got better" misses the threshold. The encoder was never an architectural commitment. It was a tax that stops paying at a certain size.