AI Inference Is Now Priced By Memory, Not Compute

AI inference is being repriced by what a chip can remember between turns, not by what it can multiply per second. When traffic looks like coding agents, with 142,000-token inputs, 444-token outputs, three-second waits while a tool runs, and subagents spawning inside a session, the question stops being who has the most FLOPs and becomes who keeps the previous turn's reusable state warm in high-bandwidth memory and routes the next turn back to it cheaply.

Forbes's Janakiram MSV, citing SemiAnalysis's new AgentX 1.0 benchmark, reports Nvidia hardware reaching up to five times better cost efficiency than AMD's MI355X on GLM 5.3 served through open-source SGLang, at 150 output tokens per second per user. That number is real for that operating point. The same SemiAnalysis publishes, elsewhere on the same model, a case where MI355X comes in 40% cheaper per million tokens on FP8 SGLang. Both numbers come from the same publisher. Both are true at the configuration measured.

The mechanism underneath: the cost gap moves with how much of each turn's KV cache can be reused from the previous turn. On DeepSeek V4 at 384 concurrent sessions, SemiAnalysis measured a 91% hit rate in HBM cache. The chip that keeps that state hot and serves the next turn from it wins. The chip that has to rebuild it loses. Forbes's headline reads as a chip verdict. The actual signal is a serving-stack verdict.