What is data skew in a distributed system like Spark, and how do you address it?
ML System DesignLevel: FoundationData pipelines and distributed processing
What is data skew in a distributed system like Spark, and how do you address it?
Hints
- Start with quality, load and latency requirements.
- Trace data to prediction; identify a failure point and a validation method.
Answer
Data skew occurs when data isn't evenly distributed across partitions — e.g., a groupBy or join key where one value (like a popular product ID) accounts for a disproportionate share of rows, causing one partition/task to take far longer than the rest and become a bottleneck (stragglers), while other executors sit idle. Mitigations: salting the skewed key (adding a random suffix to spread it across more partitions, then aggregating results afterward), using Spark's adaptive query execution (AQE) which can automatically detect and handle skewed joins, or repartitioning explicitly with a custom partitioner that accounts for the known skew.
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT