Skip to content
Quantoros

What is the difference between online and offline (batch) model inference, and when would you use each?

ML System DesignLevel: FoundationML system design and MLOps

What is the difference between online and offline (batch) model inference, and when would you use each?

Hints
  • Start with quality, load and latency requirements.
  • Trace data to prediction; identify a failure point and a validation method.

Answer

Batch inference scores a large set of examples on a schedule (e.g., nightly churn scores for all users) and stores results for later lookup — simpler infrastructure, no strict latency requirement, but predictions can be stale by the time they're used. Online inference scores individual requests in real time as they arrive (e.g., a fraud check at transaction time) — requires a low-latency serving infrastructure (model loaded in memory, feature retrieval must also be fast) but gives fresh, request-specific predictions. Choose batch when the use case tolerates staleness and you can precompute for a bounded set of entities; choose online when predictions must reflect the current request context or when the space of possible inputs is too large to precompute.

Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT

Related questions