Streaming experts make giant MoE models locally usable, with real tradeoffs

The interesting part of the new “streaming experts” demos is not that a laptop has magically become a datacenter. It is that a specific class of giant open models—Mixture-of-Experts systems that activate only a small slice of their weights per token—can now be made locally usable on Apple hardware by treating SSD bandwidth as part of the inference stack. The raised-eyebrow part is that the viral version of this story skips the engineering compromises that make it work.

The source most people saw was Simon Willison’s new post, which highlighted demos of Moonshot AI’s Kimi K2.5 on a MacBook Pro and Qwen on an iPhone. But the primary reporting trail runs through Dan Woods’ flash-moe repository, not Simon’s roundup. Woods built a custom C, Objective-C, and Metal inference stack that runs Qwen3.5-397B-A17B, Alibaba-backed Qwen’s sparse 397 billion-parameter model, on a 48 GB Apple M3 Max machine by streaming expert weights from SSD on demand.

That architecture is not a stunt pulled from nowhere. Woods explicitly builds on Apple researchers’ “LLM in a Flash” work, first described in the arXiv preprint “Efficient Large Language Model Inference with Limited Memory” and summarized on Apple’s machine learning research site. Apple’s argument was straightforward and more profound than it first sounds: when DRAM is the bottleneck, the problem is no longer just matrix math. It is memory hierarchy. Their techniques aimed to reduce how much data had to move from flash and to read larger contiguous chunks because flash storage performs badly on tiny random reads.

Woods’ repo is the more practical proof that the idea has escaped the lab. His published numbers put the 4-bit “production” configuration at 4.36 tokens per second, with the model’s 209 GB expert set streamed from SSD while about 5.5 GB of non-expert weights stay resident in memory, according to the flash-moe README. He also reports a faster 2-bit configuration at 5.74 tokens per second. But Woods says that mode degrades output quality and breaks JSON or tool-calling reliability. That is the number the meme wants; it is not the number a builder should trust.

The caveat that matters even more is architectural. According to Qwen’s model card, Qwen3.5-397B-A17B has 397 billion total parameters, 17 billion activated parameters, 512 experts, and activates 10 routed experts plus one shared expert per token. Woods’ implementation, as Simon Willison noted in an earlier post tracing the project, appears to cut that routed-expert budget to four. In other words, this is not “stock Qwen, unchanged, now casually running on a laptop.” It is a carefully tuned local inference system that trades routing budget, quantization, and some reliability to fit the hardware reality.

That distinction matters because parameter count has become a lousy shorthand for what can run locally. In sparse MoE models, the more useful question is how many parameters activate per token, how those expert weights are fetched, and whether the storage system can keep up. Woods’ repo suggests the answer on Apple Silicon is increasingly yes—if the model is sparse enough and the software stack is built around that sparsity.

Some of the most interesting lessons here are systems lessons, not model lessons. Woods writes that several more bespoke caching strategies lost to simply letting macOS page cache handle the workload, and that trying to overlap SSD transfers with GPU compute on Apple Silicon actually hurt performance rather than helping it, according to the project repository. There is something almost annoyingly elegant about that result: after a pile of clever experiments, the winning move was partly to trust the operating system.

The surrounding ecosystem suggests this is already turning into a small optimization race. Anemll’s fork of flash-moe extends the same SSD-streaming approach to newer Apple hardware, and the project’s iOS-focused branch pushes toward mobile deployment and publishes additional speed and perplexity tradeoffs across quantization settings. That does not independently verify every viral iPhone demo now circulating. But it does show the technique is spreading beyond one repo and one benchmark chart.

So what changed? Not the laws of memory. What changed is that Apple’s flash-streaming research, MoE sparsity, and a builder willing to rewrite the inference stack all met at the same moment. The result is a more serious local-AI story than the headline version suggests. Very large open models are no longer automatically trapped on datacenter-class VRAM. But they are not escaping for free. Speed, quality, routing budget, and tool reliability are all being bargained against each other, and the honest story lives in that bargain.