A new optimizer claims to cut the memory cost of training large AI 'mixture-of-experts' models by 97%

Most optimizers treat every parameter in a large "mixture-of-experts" model the same way. SkewAdam refuses to. The author reports a 97% cut in optimizer-state memory on a 6.78-billion-parameter MoE, and the saving comes from accepting a simple fact the optimizer world usually ignores: an MoE is not a homogeneous bag of parameters, so its training bookkeeping should not be one either.

The bottleneck is optimizer state, not the model itself. Training large AI systems means keeping a shadow set of values for every weight: a momentum term (the running direction past updates have been pushing) and a second-moment term (the running magnitude of those updates). For dense models, those are kept in 32-bit floats and they roughly double memory cost versus the model weights. For MoE models that split their work across many smaller sub-networks ("experts") and only activate a few per input, the parameter count is high while the per-step compute stays bounded. Optimizer state, which scales with parameter count, becomes the dominant line item, not the weights.

SkewAdam's move is to allocate state precision by parameter role instead of giving every parameter the same treatment. The backbone, about 5% of the parameters, gets full 32-bit momentum and a low-rank factored second moment (a cheaper-to-store approximation). The experts, about 95% of parameters, get only a low-rank factored second moment. The router, the small network that decides which expert handles which input and accounts for under 0.01% of parameters, gets an exact second moment. On the author's 6.78B MoE benchmark, optimizer-state memory drops from 50.6 GB to 1.29 GB, a 97.4% reduction, and peak training memory falls from 81.4 GB to 31.3 GB. A 6.78B MoE fits on a single 40 GB accelerator, where it previously needed roughly 80 GB of high-bandwidth memory to even start.

The allocation is what gives the reader a falsifier they can run themselves. The memory saving is a function of the tier split. If the backbone, experts, and router in your own MoE land anywhere near 5/95/<0.01, the headline number is in the right ballpark. If the share is more even, the saving shrinks with it. The paper isolates the contribution with a tier-ablation: removing the tiering erases the memory win, while keeping the backbone in full precision preserves the perplexity advantage.

The "beats AdamW, Muon, and Lion on validation perplexity" claim is the author's own benchmark. The paper does sweep the baselines' learning rates, but no independent lab has reproduced the perplexity lead yet. The 97.4% number is tied to one MoE architecture with a specific backbone/expert/router split. There is no head-to-head against the established memory-efficient optimizer zoo: Adafactor, 8-bit Adam, GaLore, or ZeRO-Offload. And the artifact is a preprint, not a peer-reviewed paper, announced on r/MachineLearning by the author alongside the arXiv paper and the open-source code, with a runs/ directory for anyone who wants to verify.

That is the practical posture. The split is named, the code is public, and the result is small enough to time-box a test. The convergence claim is not settled, and the memory number is an upper bound that scales with how MoE-shaped your model actually is. If the lead holds up under independent reproduction and against the memory-efficient optimizer zoo, a 40 GB GPU becomes a more honest floor for sparse-model training runs that today start at roughly twice that. The next milestone is not a press release; it is a second lab's run directory showing whether the 97% survives.