AI's Memory Shortage Has a Twist: Less Capacity Is Now More

The AI memory race has flipped. For two years, every new AI chip stacked its high-bandwidth memory taller and taller; the next generation is going shorter. The reason is simpler than a retreat: bandwidth-per-dollar now beats raw capacity for inference, the workload that defines the AI buildout.

HBM, or high-bandwidth memory, is the class of chips physically stacked on top of every modern AI accelerator. The memory tower sits next to the GPU and feeds the model when it answers a question or generates an image. More layers mean more capacity, but they also cost more per chip and push packaging yield down. The industry's 2023–2025 cycle was a race to make those towers taller, reaching 12 layers, or 12-hi, as the standard on Nvidia's Rubin and B300 accelerators, and the roadmap was meant to keep climbing toward 16-hi and beyond.

That roadmap is breaking. Nvidia's Rubin Ultra has moved to 8-hi stacks, a capacity downgrade to 192GB per GPU from the 288GB on the standard Rubin and B300, according to SemiAnalysis, the independent semiconductor research firm that first flagged the change. The firm's reading is that the shift is deliberate: for inference, where the model is mostly waiting on memory rather than computing, shorter stacks deliver the lowest cost per token of any configuration on the table.

The math depends on the workload. Training wants capacity. Pre-training a frontier model is dominated by tensor math, but the memory tower has to hold the whole model and its optimizer state in flight; if it doesn't, the GPU starves. Inference is different. Once the model is loaded, what matters is how fast tokens stream out, and that stream is gated by memory bandwidth rather than by how much memory sits on the chip. Beyond a threshold, additional HBM capacity yields diminishing returns while still carrying the full bill-of-materials cost, leaving upper-stack bits under-utilized. The wasted capacity gets called "stranded."

The DRAM shortage adds a second-order effect. HBM is consuming a growing share of total DRAM wafer capacity, and the supply isn't elastic. Hyperscalers and inference labs have been optimizing tokens per Watt under data-center power constraints, but the binding constraint has shifted to tokens per HBM wafer. If a 4-hi configuration delivers the same tokens-per-second as a 12-hi tower for inference, it consumes far fewer scarce wafers. That lever is what the next round of inference chips is being designed around.

Major-lab hardware teams are reported to want 4-hi implementations in their custom ASIC programs from HBM4 onwards, locking in the shorter stack as the inference-era default. The wafer math works for memory suppliers, at least in the short term: fewer dies per stack means more stacks per wafer, raising revenue per processed wafer even as customers pay less per chip. The win-win framing is real, but it's conditional.

Training workloads don't get the same relief. Models that genuinely need 200GB-plus of on-chip memory, including frontier-scale pre-training, long context windows, and certain mixture-of-experts topologies, still want the taller stacks. The shorter-stack thesis doesn't replace the longer one; it carves inference out of it. Suppliers that bet capex on 16-hi tooling and on customer roadmaps that didn't bend are exposed too. If hyperscalers standardize on 4-hi for HBM4 ASICs, the inventory and packaging capacity built for taller stacks becomes a depreciating bet. The "short king" framing is a punchline about a real inventory risk, not a celebration.

Through 2025 the unit of account for AI memory was gigabytes per accelerator. For inference silicon shipping in 2026 and 2027 it looks closer to bandwidth-per-dollar-per-wafer, with capacity as a constraint rather than the headline. That shift isn't visible from a spec sheet; it shows up the first time a hyperscaler explains why a smaller number on the marketing slide is the cheaper one to run.