Open-Source Text-to-Speech Hits Sub-50ms Latency at About $2 Per Million Characters

A reproducible open-source setup for text-to-speech, the AI technology that turns written text into spoken audio, just crossed two thresholds on a single NVIDIA H100 SXM GPU: sub-50 millisecond p95 time-to-first-audio (the moment a user hears the first syllable after sending a request) at 10 sustained requests per second, and roughly $2 per million characters at full utilization.

The second number is the one that changes who can ship conversational voice.

Nari Labs published the implementation, benchmark, and methodology this week. Its tuned 1.7B-parameter Qwen3-TTS CustomVoice stack can sustain 10 requests per second with audible TTFA holding below 50 milliseconds at the 95th percentile, and under 100 milliseconds even at 20 RPS. The Hacker News thread confirmed a 34 ms p95 figure on a comparable single-GPU run. Throughput lands around 630 characters per second at 10 RPS.

At a $4.29/hour H100 SXM rate, the stack burns through characters at about $2 per million. ElevenLabs' V3 lists at $100 per million characters, and Cartesia's Sonic 3.5 lists at $49 per million, per the same vendor benchmark. The open-source, self-hosted option comes in roughly 50× below ElevenLabs' list price and 25× below Cartesia's.

A startup running an always-on voice agent on ElevenLabs at $100 per million characters is paying about $0.10 per minute of speech. At $2 per million, that same minute costs about $0.002. The break-even line for "can we put a real-time voice agent in this product" just moved from a board-level decision to an engineering sprint.

The Nari benchmark defines real-time TTS with four requirements, and the threshold falls apart if any one of them is missing: low audible TTFA, zero underruns (the moment audio playback stalls because the server couldn't keep up), capacity to handle concurrent traffic, and non-malformed output. A model with a fast first-syllable response that drops packets under load isn't real-time; a model with stable throughput that takes 800 ms to start isn't either. The four-part rubric, laid out in the tts-bench methodology, is the lens the rest of the TTS conversation will be read through.

The methodology follows Fireworks AI's LLM benchmark shape: 5 minutes of Poisson open-loop traffic (request arrivals modeled as a random process with a fixed average rate, approximating how real users hit a server), a single HTTP request with streamed audio, audible TTFA reconstructed from the PCM audio stream, and Deepgram STT for evaluation. That's a synthetic workload, not a production trace, and any single-server number will move once a real call center queue lands on it.

Nari also published what the upstream engines do without its tuning. At 1 request per second, the default configurations of vLLM-Omni returned a 277.9 ms p95 TTFA with 100% underruns, SGLang-Omni 1,140.7 ms, VoxServe 315.1 ms, and M* 1,160.0 ms. These are baseline defaults at 1 RPS, not head-to-head with Nari's tuned stack. The comparison engines received only compatibility-level changes rather than matched optimization, which is why the same model on the same hardware can swing from 1.1 seconds of p95 latency to 34 ms with serving-layer work. The result isn't a leaderboard.

The benchmark is vendor-run. The ElevenLabs and Cartesia prices are list price, not negotiated or committed-use tiers. The $2-per-million figure assumes 100% H100 utilization on a single-tenant machine; mixed workloads and batching windows will land higher. A community CUDA-graph fork the Nari blog highlights is a parallel implementation, not an independent re-benchmark, so it confirms the technique is reproducible without validating the specific numbers.

The threshold doesn't make ElevenLabs or Cartesia obsolete; both still offer managed infrastructure, voice cloning, and tiered quality that an open-source stack doesn't replicate out of the box. What it does is reset the planning assumption. When a product team asks "can we add a real-time voice agent," the answer in 2026 is no longer gated on whether the API bill is survivable. It's gated on whether the team can run, monitor, and tune a single H100.

The team says a 70B-parameter run is queued for August.