What's the difference between batch processing and stream processing, and how do you decide which one to use?
ML System DesignLevel: IntermediateData pipelines and distributed processing
What's the difference between batch processing and stream processing, and how do you decide which one to use?
Hints
- Compare a finite dataset with a continuous event stream.
- Consider latency, reprocessing, late events and result consistency.
Answer
Batch processing (e.g., a nightly Spark job) processes data in large, discrete chunks on a schedule — simpler to reason about and debug, appropriate when some latency (minutes to hours) is acceptable. Stream processing (e.g., Kafka Streams, Flink, Spark Structured Streaming) processes data continuously as it arrives, enabling near-real-time results — necessary for use cases like fraud detection or real-time dashboards where minutes of delay matters. The tradeoff is stream processing is architecturally more complex (handling out-of-order events, exactly-once semantics, state management) — so unless there's a genuine business need for low latency, batch is often the simpler, more maintainable choice.
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT