AI Models Have Outgrown the Memory Inside Your Devices

An AI assistant on a phone, laptop, or car stalls mid-sentence, drops the context the user was building up, or simply refuses a request. The cause is usually not a slow chip. The device's memory architecture was designed for a different class of workload, and the industry is now rebuilding it from the inside out.

Take the Qwen 3.5 family of open models. At 8-bit quantization, a 2-billion-parameter model runs comfortably inside the ~8GB of fast discrete memory many client and edge systems still ship with. Move up to a 35-billion-parameter mixture-of-experts model, also at 8-bit, and the working set during execution pushes past 35GB. The model and its intermediate state no longer fit. The workload terminates, spills to shared system memory, or refuses to run at all. (EE Times — UMA: The Architecture Edge AI Needs to Scale)

Compute on devices has been getting cheaper, but memory has not kept up. Edge AI, which runs on a phone, laptop, or car instead of in a distant data center, was meant to put capable assistants directly in the user's hand. Many client devices still treat accelerated AI as a side path that borrows ~8GB of VRAM from a graphics subsystem originally sized for games and video. That design was tuned for predictable, bursty access patterns. AI workloads, especially the reasoning and agentic systems now reaching edge hardware, do not behave that way.

Graphics and media workloads degrade gracefully when memory runs short. A graphics workload that cannot fit in fast memory falls back to system memory and runs slower; a media workload skips frames. AI workloads do not. They break. Reasoning models and agentic systems hold long-lived context, intermediate results, and step-by-step state in memory the whole time they are running. When the working set exceeds the available pool, the model is terminated with an out-of-memory error or starts swapping aggressively and stops being useful. Memory is not a transient resource for these workloads. It is a continuous requirement.

The industry's response is a coherent memory tier that spans the CPU, the GPU, and the NPU (neural processing unit) that now ships on most client SoCs (systems on a chip). The approach, sometimes branded UMA for Universal Memory Architecture, lets the model and its working state live in a single shared memory pool rather than being copied between isolated CPU, GPU, and NPU pools.

UMA-style coherent memory fabrics exist in research and in some specialized designs, but they are not yet the default in the phones, laptops, and cars most people are buying. The piece to watch is whether the next generation of client SoCs and edge accelerators treat memory as a unified, coherent tier across accelerators, or continue to ship isolated pools that the operating system has to stitch together at runtime. The Qwen 3.5 35GB/8GB gap is the cleanest version of the problem; the architectures that close it, or fail to, will decide which AI features the next device the reader buys can actually run.