What methods for solving linear regression do you know?
Classic MLLevel: IntermediateLinear regression
What methods for solving linear regression do you know?
Hints
- Write the model using the design matrix, parameters and residual.
- Distinguish conditions for computing the estimate from conditions for inference about it.
Answer
To solve linear regression, you need to find the coefficients which minimize the sum of squared errors.
Matrix Algebra method: Let's say you have X, a matrix of features, and y, a vector with the values you want to predict. After going through the matrix algebra and minimization problem, you get this solution: .
But solving this requires you to find an inverse, which can be time-consuming, if not impossible. Luckily, there are methods like Singular Value Decomposition (SVD) or QR Decomposition that can reliably calculate this part (called the pseudo-inverse) without actually needing to find an inverse. The popular python ML library sklearn uses SVD to solve least squares.
Alternative method: Gradient Descent. See explanation below.
Source and licence: Alexey Grigorev & contributors — Data Science Interviews · CC-BY-4.0