# dimensionality reduction

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-10-29. [HTML](https://dictionaryofml.org/terms/dimred.html) · [typeset PDF](https://dictionaryofml.org/terms/dimred.pdf) · [LaTeX](https://dictionaryofml.org/terms/dimred.tex) · [Python demo](https://dictionaryofml.org/terms/dimred.py) · [notebook](https://dictionaryofml.org/terms/dimred.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Dimensionality reduction uses a map $h: \mathbb{R}^{d} \to \mathbb{R}^{d'}$ to replace a raw feature vector ${\mathbf x} = (x_{1}, \,\ldots,
\,x_{d})^{\top}$ by a shorter feature vector ${\mathbf z} = (z_{1}, \,\ldots, \,z_{d'})^{\top}$, with $d' < d$. Using fewer features lowers the effective dimension of a model, which reduces the chance of overfitting. Moreover, it lowers the amount of computation needed for model training and for inference. It also enables visualization: two features can be used as the coordinates in a scatterplot. Principal component analysis learns a linear map by minimizing the reconstruction error, while a random projection uses a linear map drawn at random, without any training. An autoencoder learns a nonlinear map represented by an artificial neural network.

## Definition

A square patch of the photograph in Fig. 2 is described by $48$ numbers, and two numbers computed from them suffice to tell sky from vegetation. Dimensionality reduction uses a map $h$, a transformation $h: \mathbb{R}^{d} \rightarrow \mathbb{R}^{d'}$ to replace the raw feature vector ${\mathbf x} = (x_{1}, \,\ldots, \,x_{d})^{\top}$ of a data point by a shorter feature vector ${\mathbf z} = (z_{1}, \,\ldots, \,z_{d'})^{\top}$, with $d' < d$ (Fig. 1). Using a shorter feature vector is beneficial in several ways:

- *Statistical benefit:* It typically reduces the chance of overfitting, as reducing the number of features often reduces the effective dimension of a model.
- *Computational benefit:* Using fewer features lowers the amount of computation needed for model training and for inference. For example, linear regression methods need to invert a matrix whose size is determined by the number of features.
- *Visualization:* A map can be learned that delivers two features $z_{1},z_{2}$, which can then be used as the coordinates in a scatterplot. Each square patch of a photograph is first averaged down to a $4 \times 4$ grid of colors, still too many numbers to plot; Fig. 2 then places each patch at the two features that principal component analysis learns for it, which keep most of the variance of the raw feature vectors.

![Figure 1 of the entry dimred](https://dictionaryofml.org/terms/fig/dimred-fig1.svg)

*Figure 1: Dimensionality reduction learns a map $h$ that sends the raw feature vector ${\mathbf x}$ of a data point, a list of $d$ numbers, to a shorter feature vector ${\mathbf z}$ of $d' < d$ numbers*

![Figure 2 of the entry dimred](https://dictionaryofml.org/terms/fig/dimred-fig2.svg)

*Figure 2: Top: the photograph of the Oetscher massif with the patch grid drawn on it; each $128 \times 128$ pixel patch is one data point. Bottom: every patch placed at the two features $z_{1}, z_{2}$ that principal component analysis learns for it. The raw feature vector of a patch is a $4 \times 4$ grid of colors, $48$ numbers; the two learned features keep $86$ percent of its variance. A patch is drawn as an open circle when its blue channel exceeds the other two by a fixed margin (sky) and as a filled triangle when its green channel does (vegetation); the remaining patches, the rock face and the mixed ones along the tree line, are small dots. Sky and vegetation lie in separate regions, with $93$ percent of them on their own side of the midpoint between the two centers. Data generated by pythondemos/dimred.py*

Dimensionality reduction methods differ in the hypothesis space used for the map, in the criterion they learn it by, and in whether the map is learned from data at all. Principal component analysis learns a linear map: the map is a matrix chosen so that the reconstruction error, the squared distance between ${\mathbf x}$ and the feature vector reconstructed from ${\mathbf z}$ by a second linear map, is minimal on average over the training set (Hotelling, 1933). An autoencoder learns a nonlinear map represented by an artificial neural network, under the same criterion (Goodfellow et al., 2016, Sect. 14.1). A random projection uses a random matrix without any training: by the Johnson–Lindenstrauss lemma, the distances between the feature vectors of the data points are approximately preserved (Vershynin, 2018). Nonlinear methods beyond the autoencoder include the manifold-learning family, among them t-SNE and UMAP, which are used mainly for visualization (Ghojogh et al., 2023; Meilă and Zhang, 2024).

**See also:** [overfitting](https://dictionaryofml.org/terms/overfitting.html), [autoencoder](https://dictionaryofml.org/terms/autoencoder.html), random projection, [principal component analysis](https://dictionaryofml.org/terms/pca.html), Johnson–Lindenstrauss lemma, [feature vector](https://dictionaryofml.org/terms/featurevec.html), [hypothesis space](https://dictionaryofml.org/terms/hypospace.html).

## References

- Hotelling (1933). Analysis of a Complex of Statistical Variables into Principal Components. *Journal of Educational Psychology*.
- Goodfellow et al. (2016). Deep Learning. *MIT Press*.
- Vershynin (2018). High-Dimensional Probability: An Introduction with Applications in Data Science. *Cambridge Univ. Press*.
- Ghojogh et al. (2023). Elements of Dimensionality Reduction and Manifold Learning. *Springer*. <https://doi.org/10.1007/978-3-031-10602-6>
- Meilă and Zhang (2024). Manifold learning: what, how, and why. *Annu. Rev. Stat. Appl.*. <https://doi.org/10.1146/annurev-statistics-040522-115238>

## Cite this entry

```bibtex
@misc{dictml_dimred,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {dimensionality reduction},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-03},
  url = {https://dictionaryofml.org/terms/dimred.html}
}
```