It fits, it loads, it stutters: testing a 27-billion-parameter AI model on the GPUs you actually own

The newest 27-billion-parameter AI model from Alibaba, Qwen 3.8 27B, fits inside the 24 GB of memory on a GeForce RTX 3090 or RTX 4090. On the same consumer cards, at 64,000 tokens of context, it generates text at roughly 26 tokens per second, less than half the speed of its own predecessor. The gap sits in the software stack underneath, not the silicon.

Tom's Hardware and the independent test lab at hardware-corner.net both published full benchmark suites for Qwen 3.8 27B, an open-weight release whose trained weights are publicly downloadable and runnable on consumer hardware. They used the same model compressed with the Q4_K Small quantization (a scheme that shrinks the 27 billion weights to roughly 17 GB), but they ran it on different driver stacks and inference engines, the software that actually serves the model to the GPU. Together the two suites cover the cards most readers would actually consider buying: Nvidia's RTX 5090, 4090, and 3090; AMD's Radeon RX 7900 XTX and Radeon AI Pro R9700; Intel's Arc Pro B70; Apple's Mac Studio with M5 Max; and the GB10-based Dell Pro Max.

VRAM, the dedicated video memory on a graphics card, used to be the gating constraint for running capable models locally. With Qwen 3.8 27B's 4-bit weights at about 17 GB, every 24 GB card on the test list can hold the model plus modest context headroom. The 32 GB RTX 5090 can push further, to roughly 128,000 tokens of in-memory context before spilling to the CPU. Beyond that, the unified-memory boxes like the Dell Pro Max with GB10 take over, trading speed for the ability to hold the entire 256,000-token context. Capacity is solved.

The binding constraint, by the data, is the inference engine and the context window. On the RTX 5090, token generation drops from 74.83 t/s at 4,000 tokens of context to 22.79 t/s at 128,000 tokens, a 70% throughput loss as the conversation grows. The cause is the KV cache, the working memory the model uses to keep track of prior tokens, which scales with context and competes with the model weights for VRAM. Comparing two GPUs at one fixed context length, the way most marketing slides do, hides this entirely.

Multi-GPU setups don't escape it. A pair of RTX 5060 Ti cards, with 16 GB apiece and 32 GB of aggregate memory, runs the model at 22.47 t/s at 4,000 tokens of context and falls to 15.55 t/s at 128,000. Doubling the cards added capacity but cut peak speed by more than half compared to a single RTX 5090. Unified-memory platforms go the other direction: the M5 Max Mac Studio, with 128 GB of shared memory, holds a 64,000-token context at 18.2 t/s, while the GB10 Dell Pro Max runs a full 256,000 tokens in memory but at 5.94 t/s, slower than human reading speed.

The second-order problem is engine support, not VRAM. Qwen 3.8 27B ships with built-in multi-token prediction, or MTP, a feature that lets the inference engine draft several tokens at once and accelerates generation when it is wired in. Tom's Hardware tested MTP both enabled and disabled and flagged a real gating variable: not every runner supports it. An RTX 5090 paired with a runner that has MTP wired in is a different product than the same card with a runner that does not, even though the silicon is identical. The model file is the same; the user experience is not.

The comparison that frames the whole suite is the new model against its own predecessor. On the RTX 5090 at 64,000 tokens of context, Qwen 3.8 27B generates 26.22 t/s versus 63.66 t/s for Qwen 3.6 27B. Prompt processing, the speed at which the model reads a long input, also falls, to 717.76 t/s from 1,957.43 t/s. Qwen 3.8 added capability (longer effective context, a more recent training mix, MTP support), and it is not faster than Qwen 3.6 at long context. The release is a quality and feature upgrade, not a speed upgrade.

The practical takeaway, drawn from the data: when a vendor markets a new model as "fits on a 24 GB card," the question is not whether the weights will load. They will. The questions that decide whether the local experience matches the demo are which inference engine runs the model, whether that engine supports features like MTP, how mature the driver stack is on the target GPU, and how much context the user actually needs. Weight size and VRAM headroom are necessary. They are no longer sufficient.

The next open-weight release will land on the same marketing treadmill. The cheap precheck, "does it fit in my GPU," will be true again. The expensive precheck, "does it run at the speed the demo implied on the work I actually do," will be the differentiator. Two benchmarkers just published the methodology to answer it.