What’s singular value decomposition? How is it typically used for machine learning?
MathematicsLevel: IntermediateDimensionality reduction
What’s singular value decomposition? How is it typically used for machine learning?
Hints
- Translate the statement into notation and check the domain.
- Start with a simple case, then check the theorem's assumptions.
Answer
For a real matrix X, singular value decomposition writes X = UΣVᵀ. Columns of U and V are orthonormal singular vectors; the diagonal entries of Σ are nonnegative singular values. Singular values are numbers, not vectors, and their squares are eigenvalues of XᵀX or XXᵀ.
For column-centered data with n rows, the PCA directions are the columns of V and their sample covariance eigenvalues are σᵢ²/(n−1). Keeping the largest k singular values gives a rank-k approximation, useful for compression, denoising and dimensionality reduction. SVD also supports stable least-squares and pseudoinverse calculations. PCA on uncentered data instead captures offsets as well as variation, so centering is a meaningful modeling choice.
Source and licence: Alexey Grigorev & contributors — Data Science Interviews · CC-BY-4.0 · answer: Quantoros editorial