Skip to content
Quantoros

Explain Whisper's architecture and its strengths over traditional ASR.

Computer VisionLevel: IntermediateAdvanced Audio Models

Explain Whisper's architecture and its strengths over traditional ASR.

Hints
  • Identify the prediction unit: image, object, pixel or sequence.
  • How do occlusion, lighting or domain changes affect results and evaluation?

Answer

Whisper (OpenAI, 2022) is a general-purpose speech recognition model trained on 680K hours of weakly-supervised audio-text pairs.

Architecture: encoder-decoder Transformer

  • Input: 30-second audio window → log-mel spectrogram (80 mel bins × 3000 time steps)
  • Encoder: ViT-like 2D convolutional layers + Transformer encoder → audio representations
  • Decoder: autoregressive Transformer decoder → text tokens (multilingual, with language token)

Key strengths over traditional ASR (DeepSpeech, Wav2Vec):

  • Robustness: trained on diverse real-world audio including noisy environments, accents, phone calls
  • Zero-shot multilingual: one model handles 99 languages without separate acoustic models
  • Timestamp generation: produces word-level timestamps useful for subtitles and search
  • Code-switching: handles mixed-language audio

Limitations:

  • Batch-only (not streaming) — 30-second chunks; latency ~0.5–2s depending on GPU
  • Long audio requires chunking with overlapping windows (word repetition at boundaries is common)
  • Hallucination: can generate plausible-sounding transcription for inaudible audio

Streaming ASR alternatives: for low-latency requirements (<200ms), use CTC-based models (wav2vec2, Conformer-CTC) which don't require waiting for full audio context.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions