What is the difference between online and offline (batch) model inference, and when would you use each?
ML System DesignLevel: FoundationML system design and MLOps
What is the difference between online and offline (batch) model inference, and when would you use each?
Hints
- Start with quality, load and latency requirements.
- Trace data to prediction; identify a failure point and a validation method.
Answer
Batch inference scores a large set of examples on a schedule (e.g., nightly churn scores for all users) and stores results for later lookup — simpler infrastructure, no strict latency requirement, but predictions can be stale by the time they're used. Online inference scores individual requests in real time as they arrive (e.g., a fraud check at transaction time) — requires a low-latency serving infrastructure (model loaded in memory, feature retrieval must also be fast) but gives fresh, request-specific predictions. Choose batch when the use case tolerates staleness and you can precompute for a bounded set of entities; choose online when predictions must reflect the current request context or when the space of possible inputs is too large to precompute.
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT