Dictionary of Applied Machine Learning
Updated on 2026-09-02
Typeset PDF version — the authoritative form of this entry
Python demo — a script that recomputes what this entry states and prints one line per check
The singular value decomposition (SVD) is a factorization of a matrix $\mA \in \reals^{\samplesize \times \featuredim}$ of the form $\mA = \mV \bm{\Lambda} \mU^{\top}$ with orthonormal matrices $\mV$ and $\mU$. The matrix $\bm{\Lambda}$ is nonzero only along its main diagonal, whose entries are nonnegative and referred to as singular values. In contrast to an eigenvalue decomposition (EVD), which only exists for a square diagonalizable matrix, an SVD exists for every matrix. Truncating the SVD after the $k$ largest singular values yields the best approximation by a matrix of rank $k$, which underlies dimensionality reduction. The SVD delivers the pseudoinverse, the spectral norm, and the condition number of a matrix.
P-existThe SVD
of a matrix $\mA \in \reals^{\samplesize \times \featuredim}$
is a factorization of the form
\[
\mA = \mV {\bm \Lambda} \mU^{\top}
\]
with orthogonal matrices (Golub and Loan, 2013)
\[
\mV = \big(\vv^{(1)},\,\ldots,\,\vv^{(\samplesize)}\big) \in \reals^{\samplesize \times \samplesize},
\qquad
\mU = \big( \vu^{(1)},\,\ldots,\,\vu^{(\featuredim)} \big) \in \reals^{\featuredim \times \featuredim}
\text{.}
\]
The matrix ${\bm \Lambda} \in \reals^{\samplesize \times \featuredim}$ is
only nonzero along the main diagonal, whose entries
$\Lambda_{\featureidx,\featureidx} = \eigval{\featureidx} \geq 0$
are referred to as singular values and ordered as
$\eigval{1} \geq \eigval{2} \geq \ldots \geq 0$. Multiplying
the factorization by $\vu^{(\featureidx)}$ gives
$\mA \vu^{(\featureidx)} = \eigval{\featureidx} \vv^{(\featureidx)}$:
$\mA$ maps the orthonormal vectors
$\vu^{(\featureidx)}$ to the scaled orthogonal
vectors $\eigval{\featureidx} \vv^{(\featureidx)}$
(see Fig. 1).
P-lowrankRetaining only the $k$ largest singular values yields the truncated SVD \[ \widehat{\mA} \defeq \sum_{\featureidx=1}^{k} \eigval{\featureidx} \vv^{(\featureidx)} \big(\vu^{(\featureidx)}\big)^{\top} \text{,} \] a matrix of rank at most $k$. Among all matrices $\mB$ of rank at most $k$, the truncation $\widehat{\mA}$ minimizes the approximation error $\normgeneric{\mA - \mB}{\mathrm{F}}$ (Eckart and Young, 1936).
The SVD can be used to compress data. A grayscale image, stored as a
matrix of $\samplesize \times \featuredim$ pixel intensities,
is compressed by keeping only the $k$ largest singular values:
the truncation requires only $k (\samplesize + \featuredim + 1)$
numbers (pythondemos/svd.py).
The SVD can be used to implement principal component analysis (PCA). For a centered feature matrix $\featuremtx \in \reals^{\samplesize \times \featuredim}$, the right singular vectors $\vu^{(\featureidx)}$ are the eigenvectors of the sample covariance matrix $\frac{1}{\samplesize} \featuremtx^{\top} \featuremtx$, whose eigenvalues are $\eigval{\featureidx}^{2}/\samplesize$. PCA projects each data point onto the leading $\vu^{(1)},\,\ldots,\,\vu^{(k)}$, the $k$ directions of largest variance, giving a linear dimensionality reduction.
P-pinvThe SVD can also be used to analyze and to implement linear regression methods. Given a feature matrix $\featuremtx \in \reals^{\samplesize \times \featuredim}$, whose $\sampleidx$-th row is the feature vector of the $\sampleidx$-th data point, and a label vector $\labelvec \in \reals^{\samplesize}$, linear regression minimizes the squared error $\normgeneric{\featuremtx \weights - \labelvec}{2}^{2}$ over the model parameters $\weights$, the empirical risk minimization (ERM) objective for the squared loss (the least squares problem). Let $k$ be the number of nonzero singular values (the rank of $\featuremtx$). A component of $\weights$ along a right singular vector $\vu^{(\featureidx)}$ with $\featureidx > k$ leaves $\featuremtx \weights$ unchanged, so expand $\weights = \sum_{\featureidx=1}^{k} \widetilde{w}^{(\featureidx)} \vu^{(\featureidx)}$ in the leading right singular vectors. Using $\featuremtx \vu^{(\featureidx)} = \eigval{\featureidx}\vv^{(\featureidx)}$ and the orthonormality of the left singular vectors $\vv^{(\featureidx)}$, \[ \begin{aligned} \normgeneric{\featuremtx \weights - \labelvec}{2}^{2} &= \normgeneric{\textstyle\sum_{\featureidx=1}^{k} \eigval{\featureidx}\widetilde{w}^{(\featureidx)} \vv^{(\featureidx)} - \labelvec}{2}^{2} \\ &= \sum_{\featureidx=1}^{k} \big(\eigval{\featureidx}\widetilde{w}^{(\featureidx)} - (\vv^{(\featureidx)})^{\top}\labelvec\big)^{2} + \sum_{\featureidx=k+1}^{\samplesize} \big((\vv^{(\featureidx)})^{\top}\labelvec\big)^{2} \text{.} \end{aligned} \] Only the components $(\vv^{(\featureidx)})^{\top}\labelvec$ of $\labelvec$ along the left singular vectors enter; the second sum is the constant, irreducible error from the part of $\labelvec$ outside the column space of $\featuremtx$. The first sum is minimized termwise at $\widehat{w}^{(\featureidx)} = (\vv^{(\featureidx)})^{\top}\labelvec / \eigval{\featureidx}$ for $\featureidx \leq k$, the minimum-Euclidean norm (pseudoinverse) solution.
See also: matrix, orthogonal, orthonormal, singular value, eigenvalue decomposition, spectral norm, condition number, rank, pseudoinverse, principal component analysis, dimensionality reduction, linear regression.
@misc{dictml_svd,
author = {Jung, Alexander},
editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
title = {singular value decomposition},
howpublished = {Dictionary of Applied Machine Learning (course edition)},
year = {2026},
doi = {10.5281/zenodo.21569296},
note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-09-21},
url = {https://dictionaryofml.org/terms/svd.html}
}