Qwen3-TTS at 10 RPS with sub-50 ms TTFA: Nari Labs pushes the speed-cost frontier

Nari Labs has published benchmarks and an open-source implementation for serving Qwen3-TTS 1.7B CustomVoice that claims a significant speed-cost win. Their stack achieves 10 requests per second (RPS) with sub-50 ms p95 time-to-first-audio (TTFA) on a single NVIDIA H100 SXM, while maintaining real-time playback with zero underruns. At full utilization, that translates to roughly $2 per 1M characters, compared to $100/1M for ElevenLabs V3 and $49/1M for Cartesia Sonic 3.5.

What “real-time” means here

The authors define real-time TTS as a four-part problem: low audible TTFA, zero underruns once playback starts, capacity that holds as RPS increases, and non-malformed output. They benchmark under Poisson open-loop traffic for five minutes, following Fireworks AI's LLM benchmark methodology. Each engine receives the full text in a single HTTP request, with audio streamed back.

Baseline performance: room for improvement

At 1 RPS with default configurations, the four compared engines show wide variance in p95 audible TTFA: vLLM-Omni at 277.9 ms, SGLang-Omni at 1,140.7 ms, VoxServe at 315.1 ms, and M* at 1,160.0 ms. All had significant leading silence (30–90 ms) and, in vLLM-Omni's case, 100% of requests had underruns.

After tuning for low latency—removing leading silence and adjusting frame accumulation—the picture changes. At 1 RPS, VoxServe hits 49.3 ms p95 TTFA, but by 6 RPS it degrades to 363.2 ms. The other engines stay above 100 ms even at 1 RPS. Nari Labs' implementation is the only one to maintain sub-50 ms p95 TTFA through 10 RPS, staying below 100 ms even at 20 RPS.

The key optimization: one scheduler for three modules

Qwen3-TTS is a three-part model: the Talker predicts the first codebook token per audio frame, the Code Predictor generates the remaining 15 tokens, and a causal Codec converts tokens to waveform samples. Most serving stacks split this into two stages—Talker+Code Predictor together, Codec separately—to overlap generation and decoding across requests.

Nari Labs instead exposes all three as independently schedulable tasks on a single scheduling surface, drawing inspiration from M*. This lets the scheduler prioritize based on urgency: before first audio, every millisecond counts; after playback starts, the next chunk only needs to arrive before the current audio finishes. The scheduler can batch requests waiting for the same module and reorder work to meet playback deadlines.

The authors argue that combining Talker and Code Predictor, while seemingly more efficient, creates a non-preemptible unit that blocks more urgent work. Keeping them separate yields shorter units and more interleaving opportunities.

Cost and open source

At $4.29/hour for a 1× H100 SXM instance (Lambda pricing), the claimed throughput of ~630 characters per second at 10 RPS works out to about $2 per 1M characters. The implementation and benchmark harness are open-sourced on GitHub.