Explain Whisper's architecture and its strengths over traditional ASR.
Computer VisionLevel: IntermediateAdvanced Audio Models
Explain Whisper's architecture and its strengths over traditional ASR.
Hints
- Identify the prediction unit: image, object, pixel or sequence.
- How do occlusion, lighting or domain changes affect results and evaluation?
Answer
Whisper (OpenAI, 2022) is a general-purpose speech recognition model trained on 680K hours of weakly-supervised audio-text pairs.
Architecture: encoder-decoder Transformer
- Input: 30-second audio window → log-mel spectrogram (80 mel bins × 3000 time steps)
- Encoder: ViT-like 2D convolutional layers + Transformer encoder → audio representations
- Decoder: autoregressive Transformer decoder → text tokens (multilingual, with language token)
Key strengths over traditional ASR (DeepSpeech, Wav2Vec):
- Robustness: trained on diverse real-world audio including noisy environments, accents, phone calls
- Zero-shot multilingual: one model handles 99 languages without separate acoustic models
- Timestamp generation: produces word-level timestamps useful for subtitles and search
- Code-switching: handles mixed-language audio
Limitations:
- Batch-only (not streaming) — 30-second chunks; latency ~0.5–2s depending on GPU
- Long audio requires chunking with overlapping windows (word repetition at boundaries is common)
- Hallucination: can generate plausible-sounding transcription for inaudible audio
Streaming ASR alternatives: for low-latency requirements (<200ms), use CTC-based models (wav2vec2, Conformer-CTC) which don't require waiting for full audio context.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0