Dictionary of Applied Machine Learning
Updated on 2026-09-18
Typeset PDF version — the authoritative form of this entry
Principal component analysis (PCA) is a dimensionality reduction method that maps the feature vectors $\featurevec \in \reals^{\featuredim}$ of a dataset to shorter vectors $\vz = \mW \featurevec \in \reals^{\featuredim'}$ by a linear feature transformation. The matrix $\mW$ is chosen so that the original feature vectors can be reconstructed from $\vz$ by a linear map with the minimum sum of squared errors over the dataset. The rows of the optimal $\mW$ are the eigenvectors of the sample covariance matrix of the centered feature vectors that belong to its $\featuredim'$ largest eigenvalues, so PCA is computed by an eigenvalue decomposition (EVD). PCA is a form of empirical risk minimization (ERM) with the squared reconstruction error as loss, and an autoencoder with linear encoder and decoder learns the same subspace. A typical application is the visualization of a dataset with thousands of features in a scatterplot of its first two principal components.
Consider a dataset \[\dataset = \big\{ \featurevec^{(1)}, \,\ldots, \,\featurevec^{(\samplesize)} \big\}\] consisting of data points characterized by feature vectors $\featurevec^{(\sampleidx)} \in \mathbb{R}^{\nrfeatures}$ for $\sampleidx=1,\,\ldots,\,\samplesize$. PCA determines, for a given number $\nrfeatures' < \nrfeatures$, a linear feature transformation \[\featuretrafovec^{(\mW)}: \mathbb{R}^{\nrfeatures} \rightarrow \mathbb{R}^{\nrfeatures'}: \featurevec \mapsto \mW \featurevec\] such that the new feature vectors $\mathbf{z}^{(\sampleidx)} = \mW \featurevec^{(\sampleidx)}$ allow reconstructing the original features with minimum linear reconstruction error (Bishop, 2006; Hastie et al., 2009; Jung, 2022): \[\min_{\mR \in \mathbb{R}^{\nrfeatures \times \nrfeatures'}} \sum_{\sampleidx=1}^{\samplesize} \normgeneric{\featurevec^{(\sampleidx)}- \mR \mW \featurevec^{(\sampleidx)}}{2}^{2}.\] PCA can be viewed as a form of empirical risk minimization (ERM) using the loss function $\lossfunc{\featurevec}{\mW} = \normgeneric{\featurevec- \widehat{\mR} \mW \featurevec}{2}^{2}$ with a reconstruction matrix $\widehat{\mR}$ that achieves the above minimum reconstruction error. It turns out that this ERM problem can be solved by a matrix $\mW = \big(\eigvecCov^{(1)},\,\ldots,\,\eigvecCov^{(\nrfeatures')}\big)^{\top}$ whose rows are given by $\nrfeatures'$ eigenvectors corresponding to the $\nrfeatures'$ largest eigenvalues of the matrix: \[\widehat{\mQ} = \frac{1}{\samplesize} \sum_{\sampleidx=1}^{\samplesize} \featurevec^{(\sampleidx)} \big( \featurevec^{(\sampleidx)}\big)^{\top} =\featuremtx^{\top} \featuremtx.\] Note that $\widehat{\mQ}$ coincides with the sample covariance matrix of $\dataset$ if its sample mean vanishes. The positive semi-definite (psd) matrix $\widehat{\mQ}$ allows for an eigenvalue decomposition (EVD) of the following form (Horn and Johnson, 2013; Meyer, 2000): \[\widehat{\mQ}\!=\!\sum_{\featureidx=1}^{\nrfeatures} \eigval{\featureidx} \eigvecCov^{(\nrfeatures)} \big( \eigvecCov^{(\nrfeatures)} \big)^{\top}= \begin{pmatrix} \eigvecCov^{(1)} & \cdots & \eigvecCov^{(\nrfeatures)} \end{pmatrix} \begin{pmatrix} \eigval{1} & & 0 \\ & \ddots & \\ 0 & & \eigval{\nrfeatures} \end{pmatrix} \begin{pmatrix} \big(\eigvecCov^{(1)}\big)^{\top} \\ \vdots \\ \big(\eigvecCov^{(\nrfeatures)}\big)^{\top} \end{pmatrix}.\] This decomposition consists of decreasing nonnegative eigenvalues $\eigval{1} \geq \eigval{2} \geq \,\ldots \,\geq \eigval{\nrfeatures} \geq 0$ and corresponding eigenvectors $\eigvecCov^{(1)},\,\ldots,\,\eigvecCov^{(\nrfeatures)}$ that form an orthonormal basis of $\mathbb{R}^{\nrfeatures}$. A common application is visualizing high-dimensional gene expression data by projecting thousands of measured gene activities onto two or three principal components that capture the dominant sources of variation.
See also: feature transformation, feature learning, dimensionality reduction, eigenvalue decomposition.
@misc{dictml_pca,
author = {Jung, Alexander and Olioumtsevits, Konstantina and Schnoor, Ekkehard},
title = {principal component analysis},
howpublished = {Dictionary of Applied Machine Learning (course edition)},
year = {2026},
doi = {10.5281/zenodo.21569296},
note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-09-19},
url = {https://dictionaryofml.org/terms/pca.html}
}