AI's Memory Problem Is Pulling Phone Chips Into the Data Center

AI servers have run out of the fast memory they were designed around. High-bandwidth memory (HBM), the stacked DRAM that sits next to a GPU and feeds it tokens at terabytes per second, is sold out through 2026, and the industry is responding by pulling low-power memory, the kind originally designed for smartphones, up the stack.

Memory has replaced compute as the binding constraint on AI. AMD said so publicly this year, and the supply chain is acting on it. TrendForce and Semianalysis both track the same squeeze, with HBM demand outpacing supply and the cost of fitting a GPU with enough of it becoming a line item on data center buildouts.

That shortage is doing two things at once. It is forcing HBM suppliers to keep expanding capacity, and it is pushing the rest of the system toward a different kind of DRAM: LPDDR, the low-power standard that has lived in phones for two decades, repackaged into a denser, swappable module called SOCAMM (small outline compression-attached memory module).

LPDDR was built for a battery. It runs cooler, draws less power, and does not need the elaborate cooling that server-grade DRAM demands. On its own, it is slower per pin than the DDR5 RDIMMs that fill today's servers. But the trade has changed. A SOCAMM module packs dense LPDDR dies into a compact, serviceable card, which gives a server more memory per socket and per watt than a traditional RDIMM. Rambus, which designs the supporting chipset, says SOCAMM draws about one-third the power of a standard DDR5 RDIMM and is easier to scale up.

That efficiency matters most at inference, the moment a trained model actually answers a question. Running a large language model is less about raw compute than about feeding it memory fast enough. Every token the model produces requires a round trip to the key-value cache (KV cache), the working memory the model uses to remember what it has already said. The bigger the model, the bigger that cache has to be, and the more often the GPU has to reach outside its own die to find it.

This is what the industry calls the memory wall: the point at which adding more compute stops speeding things up because the memory cannot keep up. Closing that gap, by placing more, faster memory closer to the CPU or GPU, is the lever that decides whether an AI workload is cheap to run or punishingly expensive.

The most visible product in this reshuffle is Micron's 256GB SOCAMM, built on the company's 1-gamma DRAM process with monolithic 32-gigabit dies. The module is sized to offload KV-cache from HBM and to improve time-to-first-token, the lag between a user's question and the model's first word. That lag is the most user-visible cost of an AI service, and shrinking it is what turns a slow demo into a product people actually use. The launch and the tradeoffs it represents are laid out in EE Times' industry analysis.

The standards bodies are following. The upcoming JEDEC LPDDR6 specification, the next generation of the low-power DRAM standard, explicitly extends its scope beyond mobile into selected data center and accelerated computing workloads. That is a consequential admission: the line between phone memory and server memory is dissolving, and the buyers driving it are hyperscale AI operators trying to make inference economics work.

LPDDR cannot replace HBM where raw bandwidth is the constraint, and SOCAMM is still a young, single-vendor ecosystem. Micron's 256GB capacity claim is a company-side number, not yet an independent teardown. And the underlying LPDDR6 standard is not finalized.

What is settled is the direction. The memory hierarchy inside an AI server is being reorganized around power and cost, not just raw speed, and the chips that win will be the ones that make every token cheaper to serve. Micron's module is one announcement. The reshuffle is the story.