Google's Qwen-on-Ironwood playbook is a bet on kernel libraries, not chips

Google's TPU engineering team published a systems engineering playbook on July 14 for running Alibaba's open-source Qwen 3.5-397B on its newest custom AI chip, Ironwood. The post reports 3.1x faster token generation and 4.7x faster prompt ingestion on Google's internal benchmarks between April and June 2026, at 512 concurrent requests.

The chips remain a moat, but the playbook treats them as a target surface. A modular, open-source-native kernel library shipped with the post, and that library is the part any other lab or cloud provider can copy. It defines what "fast inference on the largest open models" means before the next model family lands, and it positions the kernel library, not the chip itself, as the layer Google is now contesting.

Qwen, decoded. Qwen 3.5-397B is Alibaba's open-source large language model: 397 billion total parameters, but only about 17 billion active on any given token. That sparse architecture is called mixture-of-experts, and it routes each token to a small fraction of the model's weights instead of running the whole network. The full model occupies roughly 400 GB of high-bandwidth memory, which is why one inference server needs the kind of accelerator that can hold a small tree's worth of state on-chip.

Ironwood, decoded. Ironwood is Google's seventh-generation Tensor Processing Unit (TPU v7x), the silicon Google now pitches to hyperscalers and large enterprises as the alternative to Nvidia's accelerators for serving the largest open-weight models.

The methodology, decoded. Google's team decomposed a 397B-parameter MoE serving pipeline into reusable building blocks: a "Batched RPA" prefill primitive, grouped matrix multiplies, a "SparseCore" unpermutation kernel, and a new attention strategy called Attention Data Parallelism (attn-DP) that parallelizes attention across chips while the MoE expert-parallel layers communicate separately. They then ported those modules onto Qwen's specific attention variant, Gated DeltaNet (GDN), a linear-attention layer Google describes as faster and more memory-efficient than standard attention on long contexts, without rebuilding the kernels from scratch.

The reported numbers, split out: 3.1x speedup on decode-heavy workloads (the streaming-token generation a chatbot or coding assistant does once the prompt is loaded) and 4.7x on prefill-heavy workloads (the first burst of computation that pulls a long prompt into the model) at 512 concurrent requests. Both figures came from Google's own internal benchmark harness and not from an independent third party.

What makes the work portable is that the same kernels now sit in merged pull requests inside the vLLM-on-TPU repository, the open-source serving stack most production AI deployments already run on. PR #2577 wires up the correct attention data-parallelism for hybrid attention + Mamba models, the family Qwen's GDN layer belongs to. PR #2836 adds the SparseCore MoE gather and reduce kernels. PR #2679 optimizes the cross-chip reduce-scatter using a JAX primitive called psum_scatter. Each pull request is a small, audited change; together they are the receiving end of the playbook.

A second public repository, AI-Hypercomputer/tpu-recipes, holds the workload definitions and configuration. The vllm-project/tpu-inference repository itself is Apache 2.0, which means any cloud provider, lab, or enterprise can ship the same optimizations without a licensing negotiation. The upstream model artifact, Qwen/Qwen3.5-397B-A17B, is open weights from Alibaba.

The structural read. For most of the last two years, AI infrastructure competition has been a chip story: Nvidia's CUDA moat, AMD's catch-up, Google's TPU internal use, custom silicon at the hyperscalers. The Qwen-on-Ironwood playbook is a different kind of artifact. It positions the chip as a surface a kernel library can target, and the kernel library as the layer where differentiation now happens. Whoever ships the right building blocks into vLLM and SGLang, the two dominant open-source serving frameworks, gets to define what "fast inference on the largest open models" means before the next model family lands. The playbook's positioning is also a competitive argument: whoever publishes the right kernel library into the open-source serving layer first gets to set the performance standard for that model family. Both caveats — that the speedup figures are vendor-internal and that the post itself is a Google TPU pitch as much as a technical report — belong to the story.

The next quarter will tell: if the second- and third-largest open-source MoE families (the next Qwen variant and the next Llama-class release) land with first-class vLLM-on-TPU support on day one, the playbook has become the template. If they don't, the moat was the silicon all along.