A head to head CPU benchmark of four open text to speech models finds that UTMOS (an automatic voice naturalness predictor trained on listener ratings), the field's standard voice naturalness metric, stops tracking real usability for small models.
The four top open-weights text-to-speech models can now generate voice on a regular laptop CPU, and a 180-run head-to-head benchmark shows that the field's standard quality score stops tracking real usability at the bottom of the range.
A poster on r/MachineLearning tested six configurations of four models: Kokoro 82M under PyTorch and ONNX Runtime, Supertonic 3 at 2 and 5 flow-matching steps, Inflect-Nano-v1, and Pocket TTS from Kyutai Labs, all on an Intel Xeon Platinum 8272CL with 4 cores, 15.6 GB of RAM, and CUDA explicitly disabled. Every synthesis was timed with time.perf_counter() and every saved WAV was scored by UTMOS, a neural network trained to predict a 1-to-5 'mean opinion score' for how natural a synthetic voice sounds. The protocol was 6 configurations × 6 text lengths (12 to 1,712 characters) × 5 timed repetitions after a discarded warmup, totaling 180 runs; the per-run timing data and per-sample MOS are public, along with the comparison repo.
Fast and good do not co-exist in this class of model, and one config is fast to the point of unusable. Supertonic 3 in 2-step mode runs 8.3× faster than real time, with a mean RTF (wall-clock seconds per second of audio) of 0.121, but its UTMOS sits at 1.53, below the range where most listeners can follow the speech. Kokoro 82M is the opposite, with the highest UTMOS at 4.46 (PyTorch) and 4.44 (ONNX) and the slowest RTF around 0.65. Supertonic at 5 steps is the practical middle: RTF 0.24 (roughly 4× real time) with UTMOS 4.32.
The benchmark report explicitly flags that UTMOS has a known failure mode on small vocoders: it rewards outputs for being clean even when they lack prosodic naturalness. Inflect-Nano-v1 is the case in point. At 4.6 million parameters, roughly 18× smaller than Kokoro and 21× smaller than Supertonic, it scores a mid-pack UTMOS of 3.48, but by ear its output is buzzy and metallic, with the kind of HiFi-GAN texture no listener would mistake for a person. UTMOS put it in the field; listening does not. Pocket TTS scored 4.10, a hair lower, and actually sounds natural, with clean prosody and no vocoder artifacts. The score and the ear agree on Pocket TTS; they diverge on Inflect-Nano.
The right read of a UTMOS number on a small model is therefore: above the cluster at 4.3 to 4.5, you can trust the ranking; below it, treat the number as an upper bound on naturalness, not a verdict. Pairing UTMOS with a naturalness-specific score such as NISQA, or running a five-person preference test on a 100-sample subset, costs an afternoon and turns the metric from something to argue with into something to act on.
A second non-obvious finding sits in the same data. Inflect-Nano's acoustic model is configured with max_frames = 1400, which caps synthesis at about 14.93 seconds of audio regardless of input length. Inputs longer than that, anything from the long text (483 characters) upward, are silently truncated: the model returns a finished-looking WAV and the second half of the input simply isn't there. If you wire Inflect-Nano into a narration pipeline without splitting inputs into sub-15-second chunks, you ship a silent failure mode.
Pocket TTS, the newcomer in the benchmark from Paris-based Kyutai Labs, is structurally different from the rest. It is a streaming language model over Kyutai's Mimi neural audio codec, not a one-shot acoustic model plus vocoder, so its RTF is essentially flat across input length: 0.69 on a 12-character string, 0.76 on a 1,712-character essay. Kokoro climbs from 0.49 to 0.83 as inputs get longer. Supertonic 2-step drops from 0.19 on tiny inputs to 0.10 once its fixed overhead is amortized over longer synthesis. For an interactive voice system where the question is worst-case latency on cheap hardware, flat is the property worth budgeting for, not the lowest mean. The model also supports zero-shot voice cloning from roughly five seconds of reference audio, per Kyutai Labs. None of the other entrants in this benchmark ship voice cloning.
The benchmark is one CPU, one author's protocol, and a publication date of 2026-07-06. Results do not transfer to GPUs, Apple Silicon, or mobile NPUs without retesting, and the author flags reproducibility variance at ±5 to 10% across runs. Past those caveats, the surviving claim is sharper: when an objective score separates two small models by tier but listening does not, the score is the artifact that needs the disclaimer, not the model that got the lower number.