A new preprint argues that the three main efficiency levers inside transformer inference — where each token looks (attention), which specialist handles it (sparse expert routing), and how much cache memory it gets (cache precision) — are tightly
Transformer inference efficiency has, until recently, been treated as three separate problems. One team optimizes which attention pattern each token uses (full, local, or skip). Another team sparsifies the feed-forward network (FFN) by routing tokens through a small expert subset. A third team quantizes the key-value cache to shrink memory and bandwidth. Each lever is mature on its own. Optimizing them in isolation leaves quality on the table or wastes FLOPs.
A new arXiv preprint, TriRoute, argues that those three problems should be one problem. The authors' premise is that a token's three choices are tightly coupled: a token that warrants full attention may also need high-precision caching regardless of which expert processes it, and a token routed to a cheap expert is unlikely to benefit from full attention. Deciding each axis independently lets the optimization contradict itself. A token that should be cheap under a memory budget may end up expensive under an attention budget because the two decisions never saw each other.
The design is a single lightweight controller that emits, per token and per layer, a coordinated policy across three axes. First, an attention mode: skip the block, run a local window, or run full attention. Second, a sparse FFN expert set drawn from a larger pool, where the controller can also choose a null expert and skip the FFN block entirely. That null choice recovers the behavior of Mixture-of-Depths (MoD), where whole transformer blocks can be skipped, inside the routing framework. Third, a KV-cache bit-width, which lets the controller trade cache precision against memory and bandwidth per token. The architecture is conditional computation decoupled from language model quality per token of compute spent.
Training handles the discrete choices with two pieces: Gumbel-Softmax with straight-through estimation, a differentiable approximation for categorical decisions like "skip vs local vs full," for attention and cache precision; and load-balanced top-k gating for expert selection, which picks the top-scoring experts while penalizing uneven routing. Everything is trained end-to-end under a single compute budget. When the combined routing choices exceed a target FLOP or memory ceiling, the model is penalized, which lets the controller trade quality against cost on the same scale as the knobs it controls.
TriRoute's framing treats Mixture-of-Experts (FFN sparsification), Mixture-of-Depths (block skipping), and KV-cache quantization as separate research strands, each with its own objective and its own ablation tables. The ablation results claim that joint routing dominates stacked single-axis routing on the FLOPs-versus-quality curve, though, as a preprint without peer review, those claims are author-reported and have not been independently replicated.
The mechanism makes the argument portable beyond a single benchmark. A token's "right" attention span, expert, and cache precision are three views of one question about how much that token is worth spending on. When the controller sees all three views at once and optimizes them under a single budget, the frontier moves. When it sees each view separately, the budget leaks between them.
Two caveats shape how to read the numbers. First, all quantitative results, including latency, memory savings, quality deltas, and ablation outcomes, are author-reported from a single arXiv preprint. There is no third-party benchmark, no production deployment evidence, and no independent replication yet. Second, the source set is mechanism-rich but quote-thin: the analysis can stand on the design argument, but any claim about how this plays in serving stacks should wait for a confirmed deployment or an independent benchmark.
What changes in practice is the unit of optimization. Today's serving systems pick an attention mode, an expert set, and a cache precision on separate schedules, then tune them in isolation. TriRoute's contribution is the case that they are one knob, and that treating them as three is the reason the FLOP-versus-quality curve has been plateauing where it has.