What happens to our linear regression model if the column z in the data is a sum of columns x and y and some random…
Classic MLLevel: IntermediateRegularization
What happens to our linear regression model if the column z in the data is a sum of columns x and y and some random noise?
Hints
- Write the model using the design matrix, parameters and residual.
- Distinguish conditions for computing the estimate from conditions for inference about it.
Answer
If x, y and z are all input features and z is nearly x + y, the design matrix has near linear dependence. Ordinary least-squares coefficients become unstable; x and y themselves need not be highly correlated. Without noise the matrix is rank deficient and coefficient solutions need not be unique. Predictions can remain useful, but individual coefficients are unreliable. Remove redundant features, use a lower-dimensional representation, or regularize with ridge regression; fit these choices using training data only.
Source and licence: Alexey Grigorev & contributors — Data Science Interviews · CC-BY-4.0 · answer: Quantoros editorial