Skip to content
Quantoros

What’s singular value decomposition? How is it typically used for machine learning?

MathematicsLevel: IntermediateDimensionality reduction

What’s singular value decomposition? How is it typically used for machine learning?

Hints
  • Translate the statement into notation and check the domain.
  • Start with a simple case, then check the theorem's assumptions.

Answer

For a real matrix X, singular value decomposition writes X = UΣVᵀ. Columns of U and V are orthonormal singular vectors; the diagonal entries of Σ are nonnegative singular values. Singular values are numbers, not vectors, and their squares are eigenvalues of XᵀX or XXᵀ.

For column-centered data with n rows, the PCA directions are the columns of V and their sample covariance eigenvalues are σᵢ²/(n−1). Keeping the largest k singular values gives a rank-k approximation, useful for compression, denoising and dimensionality reduction. SVD also supports stable least-squares and pseudoinverse calculations. PCA on uncentered data instead captures offsets as well as variation, so centering is a meaningful modeling choice.

Source and licence: Alexey Grigorev & contributors — Data Science Interviews · CC-BY-4.0 · answer: Quantoros editorial

Related questions