Skip to content
Quantoros

What is the difference between a transformation and an action in Spark, and why does this distinction matter?

ML System DesignLevel: FoundationData pipelines and distributed processing

What is the difference between a transformation and an action in Spark, and why does this distinction matter?

Hints
  • Start with quality, load and latency requirements.
  • Trace data to prediction; identify a failure point and a validation method.

Answer

Transformations (e.g., map, filter, join) are lazy — they build up a logical execution plan (DAG) without actually computing anything. Actions (e.g., collect, count, write) trigger execution of the accumulated transformations. This matters because Spark can optimize the entire DAG before executing it (e.g., combining/reordering operations, pushing filters earlier — "predicate pushdown"), and it avoids wasted computation on transformations whose results are never actually used.

Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT

Related questions