A developer put a 304-billion-weight open-weights AI model on one AMD MI300X — and shared the recipe

A solo developer has published a fully-pinned, reproducible Docker Compose stack for running DeepSeek-V4-Flash-0731, a 304-billion-parameter open-weights language model, on a single AMD MI300X data-center GPU. Every dependency is locked by cryptographic hash, so another developer can clone the repo and reproduce the run.

The MI300X, a 2023 card with 192 GB of on-card HBM3 memory (about 2.4× the capacity of an NVIDIA H100 SXM5), holds the model's 156.67 GiB of weights with room to spare. The official vLLM recipe covers newer AMD silicon and NVIDIA hardware but explicitly excludes a single-MI300X production setup, which is the gap the repo fills.

The kernel patches cover workarounds for the MI300X's non-standard "fnuz" E4M3 FP8 math format, missing fast paths in AMD's AITER library, HIP-graph hazards in sparse MLA decode, MoE routing bugs at high concurrency, and CPU-KV synchronization. On a pinned vLLM ROCm nightly 0.26.1rc1 and AITER 0.1.19 stack, the author reports single-stream decode at 168.6 tok/s, prefill near 8,000 tok/s, and 830 tok/s aggregate across 64 concurrent streams, with 256K context validated.

The catch: this is one developer's pinned configuration, not vendor-blessed, and the community numbers are aggregate, not per-user. Per-stream median sits near 90 tok/s, enough for resale or multi-tenant use, but tight for a single developer.