The new AI bottleneck is the storage layer between memory and the GPUs

For the last few model cycles, the constraint on AI was training-scale compute: how many GPUs you could run, and for how long. At this week's RAISE Summit, that conversation changed. Solidigm, the storage subsidiary of SK hynix NAND Product Solutions Corp, and chipmaker AMD both told SiliconANGLE's theCUBE that the constraint is shifting to inference time: the work of feeding context, retrieved documents, and conversation history to a model that is planning, remembering, and acting across many steps.

A single chat prompt is a one-shot inference: the model reads a fixed input and produces a fixed output. Agentic inference is different. The model decides what to retrieve, calls tools, loops back, and keeps a running memory of where it has been. That workload does not just want a faster GPU. It wants a continuous pipeline of data sitting close enough to the chips that the GPU is never idle.

Greg Matson, SVP and head of marketing and products at Solidigm, put it directly in his theCUBE interview: storage is becoming "a whole new storage tier that's being created to extend the memory for the system." Read literally, that is a claim that a new layer is forming between DRAM and the GPU's high-bandwidth memory (HBM), the fast on-chip cache that sits closest to the silicon.

If the tier forms, the suppliers who sit in it gain a durable lever over the AI stack: the same way HBM makers now ride every model release. AMD CTO and EVP Mark Papermaster framed the same shift from the silicon side, calling agentic workloads an end-to-end process that pulls on CPUs, GPUs, adaptive computing, and networking under AMD's open-source ROCm software stack.

The numbers behind the claim are early, vendor-supplied, and worth pinning down. Solidigm's AI Central Lab, announced last October at the FarmGPU facility, reported a single-node MLPerf Storage score of 116 GB/s on its D7-PS1010 SSD, with 192 D5-P5336 drives packing 23.6 PB into 16U of rack space alongside B200 and H200 GPUs on 800 Gbps Ethernet. A separate collaboration with Metrum AI on the same testbed cut DRAM footprint by up to 57% during retrieval-augmented generation by offloading the model's key-value cache, the running state that lets an attention-based model recall prior tokens, to SSD. None of those figures have been independently re-benchmarked outside the vendor release.

Today, a long context window mostly costs HBM and DRAM, both of which are scarce and expensive. Moving the bulk of that state to dense NAND cuts the per-token bill and lets a single GPU serve longer-running agents. The savings are not theoretical: Metrum's 57% DRAM reduction, if real and repeatable, is the kind of number that resets how inference fleets are sized.

That is also why the rest of the RAISE Summit vendor stack matters. Tensordyne, Parasail, and d-Matrix are all betting on inference-time silicon. Argentum AI is pitching infrastructure financing, and Agentcy Labs with Neo4j is wrapping agent memory in a knowledge-graph frame pitched at data-sovereignty buyers. Read as a list, it is the usual vendor mosaic. Read through the storage-tier lens, it is a coordinated bet that the next constraint is at the data supply, not the GPU.

Watch the next independent benchmark. Solidigm's 116 GB/s number and Metrum's 57% DRAM reduction are the load-bearing claims of the week; if a hyperscaler or a third-party lab reproduces them, the storage tier stops being a vendor pitch and becomes an architectural fact. If they do not, the "new tier" framing will be tested against the cheaper explanation: that this is the same NAND-and-DRAM stack, working harder.