Skip to content
Quantoros

What is data skew in a distributed system like Spark, and how do you address it?

ML System DesignLevel: FoundationData pipelines and distributed processing

What is data skew in a distributed system like Spark, and how do you address it?

Hints
  • Start with quality, load and latency requirements.
  • Trace data to prediction; identify a failure point and a validation method.

Answer

Data skew occurs when data isn't evenly distributed across partitions — e.g., a groupBy or join key where one value (like a popular product ID) accounts for a disproportionate share of rows, causing one partition/task to take far longer than the rest and become a bottleneck (stragglers), while other executors sit idle. Mitigations: salting the skewed key (adding a random suffix to spread it across more partitions, then aggregating results afterward), using Spark's adaptive query execution (AQE) which can automatically detect and handle skewed joins, or repartitioning explicitly with a custom partitioner that accounts for the known skew.

Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT

Related questions