How do you monitor and ensure the reliability (SLA) of a production data pipeline?
ML System DesignLevel: IntermediateData pipelines and distributed processing
How do you monitor and ensure the reliability (SLA) of a production data pipeline?
Hints
- Start with quality, load and latency requirements.
- Trace data to prediction; identify a failure point and a validation method.
Answer
Track: freshness (is data as recent as expected — alert if a pipeline hasn't run/completed on schedule), volume (row counts within expected ranges — a sudden drop or spike often signals an upstream issue), schema/quality checks (nulls, duplicates, value ranges within expected bounds — tools like Great Expectations or dbt tests), and task-level success/failure alerting in the orchestrator (Airflow's SLA misses, retries, and failure notifications). Beyond automated checks, maintain clear runbooks for common failure modes and data lineage tracking so when something breaks downstream, you can quickly trace back to the root cause upstream.
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT