How do you build a production voice AI pipeline with low latency?
Computer VisionLevel: IntermediateAdvanced Audio Models
How do you build a production voice AI pipeline with low latency?
Hints
- Separate input preparation, prediction, queueing and data movement.
- How do per-request latency, throughput and memory change with batch size?
Answer
Target: user speaks → AI responds in < 1.5 seconds perceived latency.
Architecture:
User audio stream
↓
VAD (Voice Activity Detection)
- Detect when user stops speaking (~50ms latency)
- Use WebRTC VAD or Silero VAD
↓
Streaming ASR
- For speed: Faster-Whisper (4× faster than original)
- Or: streaming Conformer model with 200ms chunk processing
- Begin transcription while user is still speaking
↓
LLM generation (streaming)
- Start generation as soon as ASR produces first complete sentence
- Stream tokens as generated
↓
Streaming TTS
- Begin converting first sentence to speech while rest still being generated
- ElevenLabs Streaming, OpenAI TTS Streaming, or Cartesia
↓
Audio playback
- Play first audio chunk before full response is ready
Latency breakdown (optimized):
| Component | Latency |
|---|---|
| VAD end-of-speech detection | 100–300ms |
| Streaming ASR (to first token) | 200–400ms |
| LLM time to first token | 300–600ms |
| TTS to first audio chunk | 100–200ms |
| Total perceived | 700–1500ms |
Key optimization: overlap stages. Don't wait for full transcription before starting LLM; don't wait for full LLM response before starting TTS.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0