How would you handle schema evolution in a data pipeline — e.g., an upstream source adds a new column?
ML System DesignLevel: IntermediateData pipelines and distributed processing
How would you handle schema evolution in a data pipeline — e.g., an upstream source adds a new column?
Hints
- Start with quality, load and latency requirements.
- Trace data to prediction; identify a failure point and a validation method.
Answer
Treat schema changes as compatibility changes between producers, stored data and readers. For a newly added optional column, specify its type and null/default behavior, test old readers against new records and new readers against old records, and validate ingestion before rollout. Version contracts and prefer explicit selected columns in downstream consumers.
Renaming or reordering is more dangerous than adding an optional field. Iceberg tracks fields with stable IDs, so a rename does not mean a new column. Plain Parquet files plus a registry do not by themselves guarantee every reader resolves fields that way. Verify the actual table format and reader configuration. For a breaking type or meaning change, use a planned migration, potentially with parallel old/new fields, rather than silently reinterpreting historical values.
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT · answer: Quantoros editorial