What is a DAG (Directed Acyclic Graph) in the context of Airflow, and why can't it have cycles?
ML System DesignLevel: FoundationData pipelines and distributed processing
What is a DAG (Directed Acyclic Graph) in the context of Airflow, and why can't it have cycles?
Hints
- Start with quality, load and latency requirements.
- Trace data to prediction; identify a failure point and a validation method.
Answer
An Airflow DAG defines a workflow as a set of tasks with explicit dependencies (edges) — e.g., "task B runs only after task A succeeds." It must be acyclic because a cycle (A depends on B, which depends on A) would create an undefined/infinite execution order with no valid starting point — the scheduler wouldn't know which task to run first. Airflow's scheduler uses topological sorting of the DAG to determine valid execution order, which is only well-defined for acyclic graphs.
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT