Skip to content
Quantoros

What is a DAG (Directed Acyclic Graph) in the context of Airflow, and why can't it have cycles?

ML System DesignLevel: FoundationData pipelines and distributed processing

What is a DAG (Directed Acyclic Graph) in the context of Airflow, and why can't it have cycles?

Hints
  • Start with quality, load and latency requirements.
  • Trace data to prediction; identify a failure point and a validation method.

Answer

An Airflow DAG defines a workflow as a set of tasks with explicit dependencies (edges) — e.g., "task B runs only after task A succeeds." It must be acyclic because a cycle (A depends on B, which depends on A) would create an undefined/infinite execution order with no valid starting point — the scheduler wouldn't know which task to run first. Airflow's scheduler uses topological sorting of the DAG to determine valid execution order, which is only well-defined for acyclic graphs.

Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT

Related questions