HWNEWS
GPUs hit a structural latency floor that bandwidth can't fix, and a new software stack is trying to keep purpose built chips from owning the premium tier.
The first time a voice assistant pauses mid-sentence, the brain registers it as broken. The same delay, the same half-second, can read as a thoughtful pause on a phone call. AI products live or die on which side of that line they land on, and right now the GPU stack that powers most of them is structurally closer to the broken side.
The fight over who fixes that is the next phase of the AI chip war, and it is being waged in two parallel lanes. In one, OpenAI is reportedly evaluating purpose-built inference silicon from Cerebras and Groq, the kind of chips designed from the silicon up to minimize response delay. In the other, a small NVIDIA-software vendor called TileRT is making the case that the GPU you already have is fast enough, if you only write the software differently.
GPUs are bandwidth machines: an 8-GPU HGX B200 server, the standard rack-scale NVIDIA box for frontier inference, can push about 64 terabytes per second of HBM memory bandwidth, the high-bandwidth memory that sits closest to the chip. That figure roughly doubles every couple of generations. Memory latency, the delay between asking the chip for a piece of data and getting it back, has barely budged. At some point, the time spent asking becomes a bigger share of the time spent answering, and the whole stack starts to feel laggy.
Time Per Output Token, or TPOT, is the industry shorthand for how long the chip takes to spit out one word of a response once it has started. As TPOT gets small, the cost of every software interruption starts to dominate. A traditional inference engine launches a fresh GPU kernel for almost every step, and a kernel launch, plus the synchronization between steps, costs a fixed amount of time whether the model is large or small. TileRT's pitch is to compile the entire decode loop into a single persistent kernel that never exits, so the chip never has to be re-poked between tokens.
The numbers TileRT has reported are striking. On a single B200 decode server running GLM5 in FP8 (an 8-bit floating-point format that halves the memory a model occupies) at batch size 1, one user at a time, the worst case for latency, TileRT says it reaches 500 tokens per second per user. SemiAnalysis has published that same figure on its InferenceX benchmark platform. SemiAnalysis has separately reported that the same model on a GB300 NVL72, a 72-GPU rack-scale box, running "traditional inference engines" — a phrase that SemiAnalysis does not pin to a specific optimized software stack — reaches roughly 170 tokens per second per user. The theoretical memory-bandwidth roofline, SemiAnalysis's own calculation, is around 3,047 tokens per second per user on an 8-GPU HGX B200, which is why the 500 figure is real but not yet a ceiling.
That roofline is the part of the story that should make a careful reader pause. A 3× speedup over a generic baseline is not a 3× speedup over a well-tuned one. The serious GPU inference stacks today, TensorRT-LLM's LLMAPI, vLLM V1, and SGLang, all already use CUDA graphs (a way to record a sequence of GPU operations once and replay them with no launch overhead) and speculative decoding (letting a smaller draft model propose several tokens at once, which the big model then verifies in a single pass). TileRT's 3× is measured against "traditional inference engines," a phrase that the SemiAnalysis benchmark does not pin to any specific tuned stack. The honest read is that the persistent-kernel approach is a real mechanism for cutting overhead, and its actual edge against the best-tuned GPU software is a separate, unanswered question.
The demand-side test is happening now. ChatGPT, Claude, and Gemini all sell premium tiers that promise faster responses. OpenAI has framed its fastest mode around the idea of response delay that feels like "Ironman JARVIS," the cinematic always-listening assistant. TileRT has a customer reference from Xiaomi claiming 1,000 tokens per second on a trillion-parameter model. If users actually pay a recurring premium for instant-feeling AI, gross margins on the inference side get a lot fatter, and the case for purpose-built silicon gets a lot stronger.
If GPU software closes the latency gap, Cerebras, Groq, and SambaNova are left serving a smaller niche than the headlines imply. If it does not, NVIDIA keeps the premium tier by default, and the dedicated-silicon companies become the picks-and-shovels bet for the workloads the GPU stack cannot reach. The persistent-kernel approach TileRT is selling, and its pairing with vLLM-style disaggregation (separating the "read the prompt" step from the "write the answer" step onto different machines so each can be tuned for its own bottleneck), is the cleanest test of the first scenario on NVIDIA hardware that has shipped.
The interesting number to watch is not 500 tokens per second. It is the one SemiAnalysis has not yet published: TileRT against TensorRT-LLM LLMAPI with speculative decoding, on the same model, at the same batch size. Until that number lands, the GPU software stack is the bet, the dedicated chips are the bet, and the latency floor is still the thing nobody has moved.