AINEWS
Two of China's biggest open weight model families (whose trained parameters are publicly released) shipped the same DeltaNet family attention, a memory efficient long context design where the model keeps a small, fixed size state of what it has
Two of China's biggest AI model families, Moonshot's Kimi and Alibaba's Qwen, now read long documents with the same unusual math. The change is not exotic: it's what falls out when you insist that a language model keep a small, forgetful hidden state of what it has already read, instead of scanning everything it has seen to write the next word.
Both ships landed in the same product cycle. Moonshot released Kimi Linear in October 2025, with the math, weights, and reference kernels all public on the MoonshotAI/Kimi-Linear repository, and a Moonshot announcement put the change in front of a non-research audience for the first time. The doubleword.ai walkthrough also points to a recent Qwen variant using the same family of attention. Two frontier Chinese labs shipping the same unusual architecture is the signal, not any single release.
Standard transformer attention has a long-standing cost. The softmax denominator ties the current query to every earlier key, so the model has to scan everything it has read so far to write the next word. That's fine for a thousand-token prompt; it's ruinous for a million-token one. Linear attention is the family of tricks that drops the softmax and replaces it with a small fixed-size state, the same way a calculator keeps a running total instead of rewriting the tape every digit. The catch is that the running total has to stay useful, and that's what the DeltaNet family is arguing about.
That's the design constraint a walkthrough from doubleword.ai turns into a recipe. Start with softmax attention, drop the denominator to get linear attention, then ask the state to correct for stale keys and you have DeltaNet. Add a forget gate and you have Gated DeltaNet. Ask the keys to be unit-norm and the state to map from key space to value space, and you have Kimi Delta Attention, the variant both Kimi Linear and Qwen ship. The author's argument is that the equations aren't clever choices: they're what falls out of demanding that a small state do this job.
The Delta Rule paper is the published backbone for the DeltaNet step, and the Kimi Linear paper lays out the gated variant Moonshot actually ships, with the recurrent and chunkwise forms spelled out for the official Triton kernels. The doubleword.ai post is the third-party derivation that first made the constraint argument legible; a Hacker News thread picked the argument apart within hours of publication, raising the standard objections about chunkwise versus recurrent forms, gate initialization, and what the architecture gives up versus the standard recipe.
A model that keeps a small, forgetful hidden state can answer questions about the end of a long document using roughly the same compute it spent on the start, which the Kimi Linear paper presents as the difference between a research demo and a product. Frontier attention is visibly fragmenting: the standard transformer recipe isn't the only thing being shipped at the top of the open-weight tree, and the DeltaNet family is one of the live alternatives.
The open question is whether the convergence is a real fork or a temporary alignment. The Kimi Linear paper frames the design as a tradeoff, not a victory, and the community discussion reaches the same conclusion from a different direction. For now, two of the most-watched open-weight model families are betting the same way, and the math is finally public enough to argue about.