Skip to content
Quantoros

How do you build a production voice AI pipeline with low latency?

Computer VisionLevel: IntermediateAdvanced Audio Models

How do you build a production voice AI pipeline with low latency?

Hints
  • Separate input preparation, prediction, queueing and data movement.
  • How do per-request latency, throughput and memory change with batch size?

Answer

Target: user speaks → AI responds in < 1.5 seconds perceived latency.

Architecture:

User audio stream
    ↓
VAD (Voice Activity Detection)
  - Detect when user stops speaking (~50ms latency)
  - Use WebRTC VAD or Silero VAD
    ↓
Streaming ASR
  - For speed: Faster-Whisper (4× faster than original)
  - Or: streaming Conformer model with 200ms chunk processing
  - Begin transcription while user is still speaking
    ↓
LLM generation (streaming)
  - Start generation as soon as ASR produces first complete sentence
  - Stream tokens as generated
    ↓
Streaming TTS
  - Begin converting first sentence to speech while rest still being generated
  - ElevenLabs Streaming, OpenAI TTS Streaming, or Cartesia
    ↓
Audio playback
  - Play first audio chunk before full response is ready

Latency breakdown (optimized):

ComponentLatency
VAD end-of-speech detection100–300ms
Streaming ASR (to first token)200–400ms
LLM time to first token300–600ms
TTS to first audio chunk100–200ms
Total perceived700–1500ms

Key optimization: overlap stages. Don't wait for full transcription before starting LLM; don't wait for full LLM response before starting TTS.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions