The NPU Bottleneck Has Inverted: From Compute To Memory

Every AI chip in a phone or a car is being retooled around the same constraint, and it is not the one the old benchmarks measured. The neural processor that once cleared vision tasks by counting arithmetic operations per second now spends most of its cycles moving model weights and attention state in and out of on-chip memory. The binding limit has inverted, from compute to bandwidth.

This is what the inverted NPU looks like in practice: an accelerator tuned for stateless, weight-reuse-heavy convolutional networks, then asked to host transformer inference, where each token pulls fresh data from a KV cache with little math to amortize the trip. The silicon was optimized for the wrong workload. Expedera's published scaling figure, 128 TFLOPS in a single core, is the arithmetic ceiling, not the daily job; the daily job is feeding that core fast enough to keep it busy.

For consumers the takeaway is a new line to read on the next data sheet. Higher TOPS numbers describe headroom. What actually limits on-device language and vision-language models is how much data the chip can shuttle per second, how large a model fits in working memory, and how often it must reach outside to DDR. The shape repeats across phones, automotive cabins, and embedded gateways: once the workload becomes a stream of small memory lookups, raw compute becomes the underused resource.