The core idea

Most neural TTS systems tokenize audio at 50–100 Hz. That fidelity is expensive: 90 minutes of speech becomes 135,000+ tokens, far beyond a practical context window. VibeVoice's central bet is that you can tokenize far more coarsely — at 7.5 Hz — and let a diffusion model paint back the acoustic detail.

Compress the timeline, not the meaning: 7.5 Hz keeps the dialogue coherent while a diffusion head restores 24 kHz detail.

Continuous 7.5 Hz tokenizers

VibeVoice uses continuous speech tokenizers — an acoustic and a semantic stream — running at a 7.5 Hz frame rate. The low rate is what makes long-sequence processing tractable while preserving enough information to reconstruct natural speech.

Why 7.5 Hz matters

  • ~10× coarser than the 50–100 Hz industry norm.
  • 90 minutes of audio ≈ 40,500 tokens — it fits a 64K context.
  • Fewer tokens means the model spends capacity on dialogue, not acoustic minutiae.

Next-token diffusion

The generation pipeline has two stages. A large language model (a Qwen2.5 backbone) handles textual context, dialogue understanding and prosodic decisions at the token level. A diffusion head then reconstructs the fine 24 kHz acoustic detail for each step.

text + speaker refs
      |
      v
[ LLM backbone (Qwen2.5) ]  ->  semantic / prosody tokens @ 7.5 Hz
      |
      v
[ diffusion head ]          ->  24 kHz acoustic waveform

This separation of concerns is why a single model can hold a long, multi-speaker conversation together: the LLM keeps track of who is speaking and what comes next, and the diffusion head worries about how it sounds.

Stretching the design: streaming and the edge

The 2026 releases did not replace the architecture — they bent it in two directions. Both start from the same constraint: a 7.5 Hz stream is small enough that a model can afford to carry context around.

Streaming: chunks, lookahead, and its own transcript

Reading a whole hour at once is exactly what lets the offline model keep speakers straight — and exactly what a live transcript cannot do. VibeVoice-ASR-Streaming works around that by interleaving three things: a fixed-size chunk of new audio, a little lookahead audio past its edge, and the text the model has already produced. Feeding its own transcript back in is what carries speaker identity across chunk boundaries, which is why there is still no separate diarization stage.

chunk_frames = 22  @ 7.5 Hz  ->  ~2.9 s of audio per emitted segment
lookahead_frames = 4          ->  ~0.5 s peek past the chunk edge

[ audio chunk | lookahead | text so far ]
      |
      v
[ LLM ]  ->  "Speaker 1: ..."
      |
      +-->  text fed back into the next chunk

Worth reading the numbers honestly: those are the values in the shipped checkpoints, and they describe a roughly three-second cadence, not sub-second latency. It is a live transcript, not an instant one.

The edge: quantize the two halves differently

VibeVoice-ASR-BitNet takes the opposite route — same length, far less hardware. Its compression is deliberately uneven, because the two halves of the model tolerate very different things: the VAE acoustic tokenizer is quantized to INT8 with fused, SIMD-optimised kernels, while the autoregressive decoder goes all the way down to BitNet-style ternary weights. Progressive quantization-aware training keeps the accuracy cost modest.

  • 4.62 GB -> 1.58 GB, about 2.9x smaller overall.
  • Real-time factor under 1 on as few as three CPU threads — no GPU.
  • 1.6-2.3x faster than whisper.cpp at a comparable ~1.6 GB size.

What it inherits and what it can't do

Because the backbone is built on Qwen2.5, the model can inherit its base behaviours — including occasionally unexpected, biased or inaccurate output. And the design is tuned for clean, sequential dialogue: it does not handle overlapping speech, background music or sound effects.

See the Specs page for the exact numbers, or Limitations for the responsible-AI guidance.