A new open source method called USAF — Ultra Sparse Adaptive Fine Tuning — trains only the sparse expert weights and router of a Mixture of Experts model, where only a small fraction of parameters activates per input, skipping the dense updates
A 12GB consumer AMD graphics card can fine-tune a 30-billion-parameter open-weight language model, according to an implementation an independent developer published this week under an Apache 2.0 license. The model in question, Qwen3-30B-A3B, is a Mixture-of-Experts (MoE) checkpoint, a class of neural network where only a small fraction of total parameters activates on any given input. That architectural detail is what makes the result possible: fine-tuning only the active expert weights turns a 30B-parameter model into something closer to a 3-billion-parameter training problem in memory.
The developer, Reddit user tsuyu122, released the method under the name USAF (Ultra Sparse Adaptive Fine-Tuning) and described the goal plainly in a project post on r/MachineLearning: if a GPU can already run inference on an MoE model, it should also be able to fine-tune it. The author states they do not have an A100, an H100, or even an RTX 4090; they own an AMD Radeon RX 6750 XT with 12GB of VRAM running on Windows. On that hardware, the USAF README reports a 180-step fine-tuning run on Qwen3-30B-A3B completed in roughly 7.8 hours, with training loss falling from 1.43 to 1.00 and held-out perplexity improving 6%.
USAF's distinguishing move is updating only the model's expert weights and its routing layer, the small neural network that decides which expert handles which token. Existing parameter-efficient methods like LoRA, QLoRA, and DoRA leave the base model frozen and train small adapter matrices on top. The README argues that for MoE models this misses the point: the router and expert weights, not adapters, determine model behavior, and adapter-based methods cannot reach them at consumer-GPU memory budgets. The author's comparison matrix claims full fine-tuning requires 120GB or more of VRAM, LoRA and DoRA roughly 60GB, and QLoRA around 24GB; USAF, by contrast, loads in 12GB and runs on AMD hardware at all, where the README states the other methods do not.
The trade-off is not free. USAF computes gradients for about 26 million parameters per step, roughly 260 times more than a typical LoRA setup of around 100,000 parameters. On an A100 the developer estimates USAF is two to three times slower per step than LoRA, which the author frames as the cost of doing sparse training rather than adapter training. Every 50 steps USAF also runs a denser "RigL" pass to prune and re-grow the active weight set, which the README estimates adds roughly 60 seconds per pass. The point of the design, the author argues, is that on consumer hardware the comparison collapses: USAF runs, the alternatives do not load.
Qwen3-30B-A3B is a real, current checkpoint, not a placeholder: it has a public model card on Hugging Face and a public QwenLM/Qwen3 repository. A reader with a compatible AMD GPU and the patience for an overnight run can reproduce the result from the published code. The author states they are not building a business around the technique, are not selling anything, and are not pursuing commercialization; the post is labeled [P] (project) on Reddit and asks for feedback from people working with MoE models.
The caveats are visible. The benchmark numbers in the USAF README are author-reported, single-GPU, and single-developer. The comparison rows for LoRA, QLoRA, and DoRA are explicitly flagged as estimates because no public benchmarks exist at this scale on this model; the "won't load" verdicts on consumer hardware are the author's claims, not independently reproduced results. No third-party benchmark of USAF on Qwen3-30B-A3B has appeared yet. Treat the result as a reproducible signal worth verifying, not a benchmarked fact.
The practical watch item is whether anyone outside the developer runs USAF on Qwen3-30B-A3B or another open MoE checkpoint and publishes numbers. Until then, the value of the release is not that a 12GB card has matched an A100, but that an inference-capable card is now a fine-tuning-capable card for the specific class of models that are becoming the default in open-weight releases.