# Dictionary of Applied Machine Learning — full text

> Every published entry (115 terms) of the open course edition, in full: definition, math (LaTeX, macros expanded), synonyms, cross-references and references. Maintained by Alexander Jung, Aalto University. CC BY 4.0, cite as DOI 10.5281/zenodo.21569296 (ISBN 978-952-64-3013-3). Index: https://dictionaryofml.org/llms.txt. Structured form: https://dictionaryofml.org/terms.json. Per-entry markdown: https://dictionaryofml.org/terms/<key>.md

Each entry below is the file at https://dictionaryofml.org/terms/<key>.md, in alphabetical order by key.

---
# accuracy

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/accuracy.html) · [typeset PDF](https://dictionaryofml.org/terms/accuracy.pdf) · [LaTeX](https://dictionaryofml.org/terms/accuracy.tex) · [Python demo](https://dictionaryofml.org/terms/accuracy.py) · [notebook](https://dictionaryofml.org/terms/accuracy.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Accuracy is the fraction of correct predictions made by a hypothesis on a dataset with a finite label space, equal to one minus the average 0/1 loss. It is widely used as a single-number metric for classification methods. Accuracy is not suitable as an objective function for training, since the 0/1 loss is non-smooth and non-convex. Empirical risk minimization uses a surrogate loss instead, a convex or smooth upper bound on the 0/1 loss, and accuracy is reported on a validation set or a test set. On imbalanced data, accuracy alone cannot tell a classifier from a baseline that ignores the features, which calls for other means of evaluation such as the confusion matrix.

## Definition

A frost warning for Krems an der Donau is right on some days and wrong on others, and the fraction of days on which it is right is its accuracy. In general, accuracy is the fraction of correct predictions made by a hypothesis $h: \mathcal{X} \rightarrow \mathcal{Y}$ on a dataset $\mathcal{D} = \big\{ \big({\mathbf x}^{(r)},
y^{(r)} \big) \big\}_{r=1}^{m}$ of feature vectors ${\mathbf x}^{(r)}$ and labels $y^{(r)}$, each label a value $y \in \mathcal{Y}$ from a finite label space $\mathcal{Y}$ (Goodfellow et al., 2016, Sect. 5.1.2; Shalev-Shwartz and Ben-David, 2014, Sect. 2.2). With the 0/1 loss $L^{(0/1)}\left({\mathbf z},h \right)$, which is $1$ for a wrong prediction on the data point ${\mathbf z}$ and $0$ otherwise, accuracy is

\begin{align*}
\operatorname{acc}(h \mid \mathcal{D})
& := 1 - \frac{1}{m}
\sum_{r=1}^{m}
L^{(0/1)}\left(\big({\mathbf x}^{(r)},
y^{(r)}\big),h \right) \\
& = \frac{1}{m}
\Big| \Big\{ r \in \{1, \,\ldots, \,m\} :
h\big({\mathbf x}^{(r)}\big)
= y^{(r)} \Big\} \Big|
\text{.}
\end{align*}

Accuracy ranges from $0$ (no prediction is correct) to $1$ (every prediction is correct). Equivalently, accuracy equals one minus the average 0/1 loss on the dataset. For example, in image classification with the label space $\mathcal{Y} = \{\text{cat}, \text{dog},
\text{bird}\}$, the hypothesis assigns each image, represented by its feature vector ${\mathbf x}$, a category $h({\mathbf x})$. Accuracy is then the fraction of images whose assigned category equals the true label $y$: $94$ correct assignments among $100$ images give an accuracy of $0.94$.

Fig. 1 compares two classifiers on the same training set $\mathcal{D}^{(\rm train)}$ of $60$ data points, the dataset from which both were learned. The linear classifier $h$, whose decision boundary is a line, misclassifies three data points near that line. The nonlinear classifier $h'$ curves around each of these data points and classifies them correctly.

![Figure 1 of the entry accuracy](https://dictionaryofml.org/terms/fig/accuracy-fig1.svg)

*Figure 1: Two classifiers applied to the same training set $\mathcal{D}^{(\rm train)}$ of $m=60$ data points with classes $y \in \{0,1\}$ (blue circles and red squares); $20$ of them are drawn, every third of the correctly classified ones and the three misclassified ones. The linear classifier $h$ (dashed line) places three data points on the wrong side of its decision boundary; they are ringed in black and lie close to the dashed line, and they yield $\operatorname{acc}(h \mid \mathcal{D}^{(\rm train)}) = 57/60 = 0.95$. The nonlinear classifier $h'$ (solid green curve) bends around those three data points and yields $\operatorname{acc}(h' \mid \mathcal{D}^{(\rm train)}) = 1$. Data generated by pythondemos/accuracy.py*

The classifier $h'$ in Fig. 1 achieves the optimal accuracy of $1$ on $\mathcal{D}^{(\rm train)}$: its decision boundary closely follows the positions of the individual data points that the linear classifier misclassifies. The classifier $h'$ is therefore sensitive to small perturbations of the feature vectors, such as measurement noise. A slight shift in the feature vector of a data point near the decision boundary can flip the prediction. Optimizing accuracy on the training set therefore does not by itself ensure good generalization (Shalev-Shwartz and Ben-David, 2014, Sect. 2.2.1). Good generalization means a small loss on data points outside the training set. Accuracy is often used as a single-number metric to compare learned classifiers: on a validation set $\mathcal{D}^{(\rm val)}$, to select among candidate hypotheses during validation, and on a test set $\mathcal{D}^{(\rm test)}$, whose data points entered neither training nor selection, to assess the selected hypothesis.

Accuracy could also be used as the objective function for training. However, accuracy is defined via the 0/1 loss, which is piecewise constant as a function of the model parameters and neither smooth nor convex (Shalev-Shwartz and Ben-David, 2014, Sect. 12.3), so accuracy is piecewise constant too: the gradient of accuracy is zero wherever it exists and therefore indicates no direction of improvement. Gradient-based optimization methods cannot maximize accuracy directly. Instead, empirical risk minimization-based methods use a surrogate loss that upper bounds the 0/1 loss and is chosen convex or smooth, so that those methods have a gradient to follow (Shalev-Shwartz and Ben-David, 2014, Sect. 12.3). Two examples for such a surrogate loss are the logistic loss, which is convex (Shalev-Shwartz and Ben-David, 2014, Sect. 9.3) and smooth (Shalev-Shwartz and Ben-David, 2014, Example 12.6), and the hinge loss, which is convex (Shalev-Shwartz and Ben-David, 2014, Sect. 12.3) but non-smooth: its one-sided derivatives at margin $1$ are $-1$ and $0$, so it is not differentiable there.

Fig. 2 shows the three loss functions as functions of the margin, for binary classification with $\mathcal{Y} = \{-1, 1\}$, where a real-valued hypothesis $h$ predicts via the sign of $h({\mathbf x})$. Each loss function depends on the data point only through the margin $y \cdot h({\mathbf x})$, which is how loss functions for binary classification are compared (Hastie et al., 2009, Sect. 10.6). The hinge loss lies above the 0/1 loss everywhere (Shalev-Shwartz and Ben-David, 2014, Sect. 12.3); the logistic loss equals $\ln 2$ at margin $0$, so it is the logistic loss scaled by $1 / \ln 2$ that lies above the 0/1 loss.

![Figure 2 of the entry accuracy](https://dictionaryofml.org/terms/fig/accuracy-fig2.svg)

*Figure 2: Three loss functions for binary classification as functions of the margin $y \cdot h({\mathbf x})$: the 0/1 loss (blue step; the filled dot marks its value $1$ at margin $0$, the open dot the value it does not take), the hinge loss (red curve), and the logistic loss scaled by $1/\ln 2$ (green curve). All three pass through $1$ at margin $0$. Both surrogates are convex, lie above the 0/1 loss, and penalize small margins where the 0/1 loss is already $0$; the two zero branches are drawn slightly above the axis so that they stay visible*

Accuracy is the metric of choice when the classes are balanced and every error costs the same; on imbalanced data it is not. Consider a frost warning for Krems an der Donau (Austria): the features of a data point are the minimum and the maximum air temperature of one day, and its label indicates whether the minimum temperature of the following day falls below $0\,^{\circ}\mathrm{C}$. Frost days are the minority class: of a training set of $40$ recorded days, only $6$ are followed by frost.

The constant prediction ``above zero'' (a baseline that ignores the features) is therefore correct on $34$ of the $40$ days, an accuracy of $0.85$, and a classifier learned by logistic regression from the $40$ data points reaches the same accuracy of $0.85$. Both make the same number of errors, so accuracy cannot tell the two apart; they differ in where they are right, which calls for other means of evaluation such as the confusion matrix. The confusion matrices in Fig. 3 show that the learned classifier detects one of the six frost days and raises one false alarm, while the baseline detects none and raises none.

![Figure 3 of the entry accuracy](https://dictionaryofml.org/terms/fig/accuracy-fig3.svg)

*Figure 3: The confusion matrices of the frost-warning classifier learned by logistic regression (a) and of the always-above-zero baseline (b). The shaded diagonal cells hold the correct predictions: $33 + 1 = 34$ for the learned classifier and $34 + 0 = 34$ for the baseline, so both reach the same accuracy of $0.85$ with $6$ errors each. The cells differ in where the correct and the wrong predictions fall: the learned classifier detects one frost day and raises one false alarm, while the baseline detects none and raises none. Data generated by pythondemos/cm.py*

**See also:** $\bf 0/1$ loss, [loss](https://dictionaryofml.org/terms/loss.html), [classification](https://dictionaryofml.org/terms/classification.html), F$_1$ score, [confusion matrix](https://dictionaryofml.org/terms/cm.html), imbalanced data.

## References

- Goodfellow et al. (2016). Deep Learning. *MIT Press*.
- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>

## Cite this entry

```bibtex
@misc{dictml_accuracy,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {accuracy},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/accuracy.html}
}
```


---

# artificial neural network (ANN)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/ann.html) · [typeset PDF](https://dictionaryofml.org/terms/ann.pdf) · [LaTeX](https://dictionaryofml.org/terms/ann.tex) · [Python demo](https://dictionaryofml.org/terms/ann.py) · [notebook](https://dictionaryofml.org/terms/ann.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

An artificial neural network (ANN) is a hypothesis space whose hypothesis maps are all built on one directed graph. Each hypothesis in this space is a map from the features of a data point to a prediction for the label, obtained for a particular choice of the learnable model parameters carried by the graph's edges (edge weights) and nodes (offset terms). Each node is an artificial neuron that applies an activation function to the weighted sum of its inputs plus its offset. The connectivity pattern, the architecture, is an important design choice: a feedforward network uses a directed acyclic graph, whereas a recurrent neural network allows a cycle. Depending on the connectivity, ANNs include deep nets, CNNs, RNNs, and transformer-based architectures.

## Definition

The face recognition of a smartphone camera and the speech transcription of a voice assistant are computed by ANNs whose connection strengths were learned from data (Goodfellow et al., 2016, Sects. 12.2 and 12.3). An ANN is a hypothesis space whose hypothesis maps are all built on one directed graph: its nodes are the artificial neurons, elementary computational units, and its weighted edges fix which artificial neurons feed which. The edge weights and the offset terms assigned to the nodes form the learnable model parameters ${\mathbf w}$ of an ANN, and a choice of ${\mathbf w}$ selects one hypothesis map $h^{({\mathbf w})}$, from the features of a data point at the input to an output used as the prediction for the label. The learned hypothesis $\widehat{h} = h^{(\widehat{{\mathbf w}})}$ is one such choice.

Each artificial neuron applies an activation function $\sigma(\cdot)$ to the weighted sum of its inputs plus an offset term (Fig. 1). The activation function can itself carry model parameters (parametric activation functions, He et al., 2015), but it is customarily prescribed in advance, for example the rectified linear unit (Goodfellow et al., 2016, Sect. 6.3), so that only the node-wise offset and the edge weights are learned.

![Figure 1 of the entry ann](https://dictionaryofml.org/terms/fig/ann-fig1.svg)

*Figure 1: A single artificial neuron: it applies an activation function $\sigma(\cdot)$ to the weighted sum $w_{1} x_1 + w_{2} x_2$ of its inputs plus an offset term $b$, and delivers the output $a_{1}$. The edge weights $w_{1}$, $w_{2}$ and the offset $b$ are learnable model parameters*

The output of a neuron can be used either as the final output of the ANN or as an input to other neurons. The directed graph of the ANN records these uses: its nodes are the artificial neurons together with the input features, and each directed edge carries the output of one node to the input of another (Fig. 2). Which outputs are connected to which downstream neuron's inputs, the connectivity structure (or architecture) of the ANN, is therefore an important design aspect (Goodfellow et al., 2016). Whether the graph is acyclic or contains a cycle separates the two broad families of ANN.

![Figure 2 of the entry ann](https://dictionaryofml.org/terms/fig/ann-fig2.svg)

*Figure 2: An ANN represented as a weighted, acyclic directed graph with nodes corresponding to neurons or features of a data point. Feature nodes $x_1$, $x_2$ have a fixed output given by the feature value, with no input. Each of the neuron nodes $a_{1}$, $a_{2}$ applies an activation function to the weighted sum of its inputs (each as in Fig. 1, whose offset terms are omitted here). The weighted directed edges, with arrowheads marking the direction of signal flow, indicate how node outputs are used as inputs to downstream neurons. The edge weights are tunable model parameters and are used to scale the inputs to the neurons. The prediction $h^{({\mathbf w})}({\mathbf x})$ is the final output of the ANN, computed from the inputs $x_1$, $a_1$, and $a_2$*

The graph of Fig. 2 is acyclic, so this ANN is feedforward (Goodfellow et al., 2016, Ch. 6): the signal passes from the inputs to the output without returning to any node. One widely used type of ANN is deep nets where neurons form consecutive layers (Fig. 3). In a deep net, the outputs of neurons in a given layer are typically only connected to the inputs of the neurons in a consecutive layer. By contrast, in the general directed acyclic graph of Fig. 2 an output may feed any downstream neuron, and there $x_1$ connects straight to the prediction.

Each layer $l$ then applies a feature transformation ${\mathbf \Phi}^{(l)}$ whose components are computed by the neurons of that layer (each as in Fig. 1), and the deep net computes the concatenation of these layer-wise maps. A shortcut or skip connection directly connects the outputs of neurons in one layer to the inputs of neurons in a nonconsecutive layer (Goodfellow et al., 2016; He et al., 2016); Fig. 3 shows one as a dashed edge. With that edge removed, the network of Fig. 3 delivers the prediction

$$
h^{({\mathbf w})}({\mathbf x})
= {\mathbf \Phi}^{(3)}\Big({\mathbf \Phi}^{(2)}
\big({\mathbf \Phi}^{(1)}({\mathbf x})\big)\Big)
\text{.}
$$

Here, the input is the feature vector ${\mathbf x} = (x_1, x_2, x_3)^{\top}$. The two hidden layers compute ${\mathbf \Phi}^{(1)}, {\mathbf \Phi}^{(2)}:
\mathbb{R}^{3} \rightarrow \mathbb{R}^{3}$, and the output layer computes ${\mathbf \Phi}^{(3)}: \mathbb{R}^{3} \rightarrow
\mathbb{R}$.

![Figure 3 of the entry ann](https://dictionaryofml.org/terms/fig/ann-fig3.svg)

*Figure 3: A deep net: the neurons form consecutive layers, and each neuron receives only the outputs of the preceding layer. Every solid edge carries a learnable edge weight (not drawn). The dashed edge is a skip connection, directly connecting two nonconsecutive layers*

The graphs seen so far are acyclic. A recurrent neural network instead includes a feedback edge (Goodfellow et al., 2016, Ch. 10), so its graph contains a cycle and is not a directed acyclic graph (Fig. 4): a self-loop feeds a neuron's output back to its own input. This feedback lets the network carry information across the steps of a sequence; unrolling the recurrence over the sequence, one copy of each neuron per step, recovers a directed acyclic graph (Goodfellow et al., 2016, Sect. 10.1).

![Figure 4 of the entry ann](https://dictionaryofml.org/terms/fig/ann-fig4.svg)

*Figure 4: A simple recurrent neural network. The self-loop on the hidden neuron $h$ (edge weight $w_{\mathrm{rec}}$) feeds its output back to its own input, so the graph contains a cycle and is not a directed acyclic graph. The feedback carries information across the steps of a sequence*

The networks of Figs. 2, 3, and 4 differ only in which nodes their directed edges join. This connectivity pattern names the common architectures. Feedforward networks have an acyclic graph, a directed acyclic graph (deep nets are the layered case), while CNNs are also feedforward but share weights across spatial positions. RNNs add feedback connections for sequence processing, and transformer-based architectures, which underlie modern LLMs, are likewise feedforward and use attention to connect every token to every other (Vaswani et al., 2017, Sect. 3). Skip connections appear in these architectures as the residual connections of deep CNNs (He et al., 2016) and transformers (Vaswani et al., 2017). Each architecture fixes a different hypothesis space, within which empirical risk minimization learns a single hypothesis $\widehat{h} = h^{(\widehat{{\mathbf w}})}$ by choosing the model parameters $\widehat{{\mathbf w}}$ that minimize the average loss on a training set. The architecture also governs the generalization of the learned hypothesis to data points outside the training set, which is what the choice of an architecture is judged by. That choice need not be made by hand: neural architecture search learns a useful connectivity automatically (Elsken et al., 2019).

**See also:** [label](https://dictionaryofml.org/terms/label.html), artificial neuron, activation function, deep net, layer, convolutional neural network, recurrent neural network, [transformer](https://dictionaryofml.org/terms/transformer.html).

## References

- Goodfellow et al. (2016). Deep Learning. *MIT Press*.
- He et al. (2015). Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. *2015 IEEE Int. Conf. Comput. Vis. (ICCV)*. <https://doi.org/10.1109/ICCV.2015.123>
- He et al. (2016). Deep Residual Learning for Image Recognition. *2016 IEEE Conf. Comput. Vis. Pattern Recognit.*. <https://doi.org/10.1109/CVPR.2016.90>
- Vaswani et al. (2017). Attention is all you need. *Adv. Neural Inf. Process. Syst.*. <https://papers.nips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html>
- Elsken et al. (2019). Neural Architecture Search: A Survey. *J. Mach. Learn. Res.*.

## Cite this entry

```bibtex
@misc{dictml_ann,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {artificial neural network (ANN)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/ann.html}
}
```


---

# attention

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/attention.html) · [typeset PDF](https://dictionaryofml.org/terms/attention.pdf) · [LaTeX](https://dictionaryofml.org/terms/attention.tex) · [Python demo](https://dictionaryofml.org/terms/attention.py) · [notebook](https://dictionaryofml.org/terms/attention.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Attention is a mechanism that models the dependencies between the tokens that make up a data point, such as the words in a sentence or the pixel patches of an image. Attention represents the relationship between two tokens by a parameterized weight function whose model parameters are learned from a training set. An attention head then computes a new vector representation for each token as a weighted combination of the value vectors of all tokens. Like the weight function, the value vectors are learned. An attention head acts as a differentiable associative memory: a token uses its query vector to retrieve, through the key vectors of the other tokens, the values most relevant to it. Attention heads represent dependencies between tokens regardless of their positions within a data point, and they are a core component of modern LLMs.

## Definition

A sentence consists of words, an image of pixel patches, and a network of nodes: the data points of some machine learning applications are composed of smaller units, referred to as tokens. In general, the tokens that constitute a single data point are not independent of one another. Instead, the meaning of a token depends on a few other tokens of the same data point, which may lie far from it. The attention mechanism is a building block of ANNs that represents such dependencies between tokens regardless of their distance within a data point (Vaswani et al., 2017, Sect. 4), and it is a core component of the LLMs built on the transformer (Brown et al., 2020).

Probabilistic models represent and analyze such dependencies (Blei et al., 2003). Attention mechanisms instead represent these dependencies directly, without specifying a probability distribution over the tokens. They represent the relationship between a token with index $i$ and a token with index $i'$ by an attention weight $f^{({\mathbf w})}(i,i')$, a parameterized function whose model parameters ${\mathbf w}$ are learned from a training set. The attention weight measures how strongly token $i$ attends to token $i'$. Practical attention mechanisms differ in their choice of the function $f^{({\mathbf w})}(i,i')$ and in the empirical risk minimization variant used to learn ${\mathbf w}$. The most widely used choice, described below, is scaled dot-product attention (Vaswani et al., 2017).

Scaled dot-product attention derives three vectors from the embedding vector ${\mathbf x}^{(i)} \in \mathbb{R}^{d}$ of each token $i$ via learned matrices. They are a query (vector) ${\mathbf q}^{(i)} = {\mathbf W}_Q\, {\mathbf x}^{(i)}$, a key (vector) ${\mathbf k}^{(i)} = {\mathbf W}_K\, {\mathbf x}^{(i)}$, and a value (vector) ${\mathbf v}^{(i)} = {\mathbf W}_V\, {\mathbf x}^{(i)}$, with the matrices

$$
{\mathbf W}_Q, {\mathbf W}_K, {\mathbf W}_V \in \mathbb{R}^{d \times d}
$$

as model parameters learned during training. The query of a token determines which keys it matches, the key of a token determines which queries match it, and the value carries the information that is aggregated (Bishop and Bishop, 2024; Elhage et al., 2021).

The output of the attention mechanism for token $i$ is a weighted sum of the values of all tokens,

$$
{\mathbf z}^{(i)}
= \sum_{i'=1}^{n} \alpha_{i,i'}\,
{\mathbf v}^{(i')} \text{.}
$$

Here, $n$ denotes the number of tokens of the data point. The coefficient $\alpha_{i,i'}$ is the attention weight, i.e., the scaled dot-product form of $f^{({\mathbf w})}(i,i')$. The attention weights are computed from the queries and keys in two steps. First, the attention score between tokens $i$ and $i'$ is

$$
s_{i,i'}
= \frac{\big({\mathbf q}^{(i)}\big)^{\top} {\mathbf k}^{(i')}}
{\sqrt{d}} \text{.}
$$

The inner product (also called dot product) $\big({\mathbf q}^{(i)}\big)^{\top} {\mathbf k}^{(i')}$ measures how well the key of token $i'$ matches the query of token $i$. Second, the attention weight $\alpha_{i,i'}$ follows by applying the softmax function to the scores $s_{i,i'}$ across all tokens $i'$:

$$
\alpha_{i,i'}
= \frac{\exp(s_{i,i'})}
{\sum_{j=1}^{n} \exp(s_{i,j})} \text{.}
$$

The softmax function normalizes the scores. The attention weights $\alpha_{i,i'}$ are nonnegative and sum to one across $i' = 1, \ldots, n$, forming a probability distribution over the tokens. Dividing the inner product by $\sqrt{d}$ in the attention score counteracts its growth with the dimension $d $. The score $\big({\mathbf q}^{(i)}\big)^{\top} {\mathbf k}^{(i')}$ is a sum of $d $ terms. Without the scaling, a large $d $ would therefore push the softmax function into a region where its gradients nearly vanish (Vaswani et al., 2017, Sect. 3.2.1). Fig. 1 illustrates scaled dot-product attention for an eight-token sentence.

![Figure 1 of the entry attention](https://dictionaryofml.org/terms/fig/attention-fig1.svg)

*Figure 1: Scaled dot-product attention for the sentence ``All human beings are born free and equal.'' Each token embedding ${\mathbf x}^{(i)}$ is mapped by learned matrices ${\mathbf W}_Q$, ${\mathbf W}_K$, ${\mathbf W}_V$ into query, key, and value vectors (boxes with solid, dashed, and dotted borders). First, the attention score $s_{i,i'}$ is computed as the scaled dot product of query $i$ and key $i'$. Then, the softmax function normalizes these scores into attention weights $\alpha_{i,i'}$ that sum to one across all positions. Arcs at the top illustrate how token 8 (``equal'') attends to earlier tokens; thicker arcs indicate higher attention weights (Vaswani et al., 2017). The token ``equal'' and its query ${\mathbf q}^{(8)}$ are highlighted by a darker fill: the depicted weights $\alpha_{8,i'}$ involve the query of token 8 but the keys and values of all tokens*

An attention head moves from a single token to the whole sequence of tokens. It is a parameterized function $h^{({\mathbf w})}$ that maps the input embeddings to the output vectors,

$$
h^{({\mathbf w})}\big({\mathbf x}^{(1)}, \ldots, {\mathbf x}^{(n)}\big)
= \big({\mathbf z}^{(1)}, \ldots, {\mathbf z}^{(n)}\big) \text{,}
$$

with model parameters ${\mathbf w}$ comprising the three matrices ${\mathbf W}_Q$, ${\mathbf W}_K$, ${\mathbf W}_V$. Collect the output vectors as the rows of a matrix ${\mathbf Z}$, and the queries, keys, and values as the rows of $\mathbf{Q}$, $\mathbf{K}$, $\mathbf{V}$. Then $h^{({\mathbf w})}$ computes them in the compact form

$$
{\mathbf Z} = \operatorname{softmax}\!\!\left(\frac{\mathbf{Q}\,\mathbf{K}^{\top}}{\sqrt{d}}\right) \mathbf{V} \text{.}
$$

Here, the softmax function is applied independently to each row of $\mathbf{Q}\,\mathbf{K}^{\top}/\sqrt{d}$, so that each row of the resulting matrix is a probability distribution over the $n$ tokens. Since the queries, keys, and values are all derived from the same sequence of tokens, this construction is called self-attention.

In practice, several attention heads, each with its own learned matrices ${\mathbf W}_Q$, ${\mathbf W}_K$, ${\mathbf W}_V$, operate in parallel; this is called multi-head attention, and each head can specialize to a different relation between tokens (Vaswani et al., 2017).

Attention can be interpreted as a differentiable associative memory, i.e., a content-addressable store (Ramsauer et al., 2021, Sect. 2, Eq. (3) and Eq. (10)): the keys ${\mathbf k}^{(i')}$ act as addresses, the values ${\mathbf v}^{(i')}$ as stored contents, and a token $i$ searches the other tokens with its query ${\mathbf q}^{(i)}$. The inner product $\big({\mathbf q}^{(i)}\big)^{\top}{\mathbf k}^{(i')}$ measures how well the address $i'$ matches the query, the softmax function turns the match scores into retrieval weights, and the output ${\mathbf z}^{(i)}$ is the retrieved content.

A classical dictionary data structure or hash table returns the single value stored at an exactly matching address (Cormen et al., 2022, Ch. 11). Attention instead returns a weighted average. Moreover, its output ${\mathbf z}^{(i)}$ is a differentiable function of the matrices ${\mathbf W}_Q$, ${\mathbf W}_K$, and ${\mathbf W}_V$. In particular, the queries, keys, and values can themselves be learned via gradient-based methods for solving empirical risk minimization. The same retrieval principle, with a weighted average in place of a single best match, is used by k-nearest neighbors methods. Computing all pairwise scores $s_{i,i'}$ requires $n^{2}$ query–key inner products, one for each ordered pair of tokens. The retrieval principle also guides efficient implementations: locality-sensitive hashing restricts each query to nearby keys, reducing the cost of one attention head from $n^{2}$ query–key comparisons to a constant multiple of $n \ln n$, with $n$ again the number of tokens (Kitaev et al., 2020, Sect. 1).

Because ${\mathbf q}^{(i)}$ and ${\mathbf k}^{(i')}$ are obtained from the different matrices ${\mathbf W}_Q$ and ${\mathbf W}_K$, the attention weights are *directed*: in general, $\alpha_{i,i'} \neq \alpha_{i',i}$, so that token $i$ attending strongly to token $i'$ does not imply the reverse.

Fig. 2 visualizes the attention weights $\alpha_{i,i'}$ that a single head learns on sentences from the Universal Declaration of Human Rights (United Nations General Assembly, 1948). Trained to reconstruct each masked token from the others, the head gives many query tokens a weight concentrated on a few keys rather than spread uniformly, illustrating the associative-memory reading on this small corpus. The heat map also shows the directed nature of the weights discussed above: the query token ``rights'' places its full weight on the key token ``beings'' ($\alpha_{12,3} = 1.0$), while the query token ``beings'' places no weight on the key token ``rights'' ($\alpha_{3,12} = 0.0$).

![Figure 2 of the entry attention](https://dictionaryofml.org/terms/fig/attention-fig2.svg)

*Figure 2: Attention weights $\alpha_{i,i'}$ of one self-attention head for the sentence ``All human beings are born free and equal in dignity and rights'' (Article 1 of the Universal Declaration of Human Rights, United Nations General Assembly, 1948). Row $i$ is the query token; column $i'$ is the searched key token. Darker cells carry more weight: for instance, the query token ``all'' places nearly all of its weight on the key token ``are'' ($\alpha_{1,4} = 0.94$), and its remaining weights are close to zero, since the weights of each query sum to one. The head was trained on sentences from the Universal Declaration of Human Rights by masking each token and reconstructing it from the others, so the query must retrieve associated tokens through their keys. Data generated by pythondemos/attention.py*

**Synonyms:** attention mechanism, attention head.

**See also:** token, embedding, [transformer](https://dictionaryofml.org/terms/transformer.html), softmax function, [large language model](https://dictionaryofml.org/terms/llm.html), $k$-nearest neighbors.

## References

- Vaswani et al. (2017). Attention is all you need. *Adv. Neural Inf. Process. Syst.*. <https://papers.nips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html>
- United Nations General Assembly (1948). Universal Declaration of Human Rights (UDHR). *United Nations*. <https://www.un.org/en/about-us/universal-declaration-of-human-rights>
- Brown et al. (2020). Language Models are Few-Shot Learners. *Adv. Neural Inf. Process. Syst.*. <https://proceedings.neurips.cc/paper_files/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html>
- Blei et al. (2003). Latent Dirichlet Allocation. *J. Mach. Learn. Res.*. <https://doi.org/10.5555/944919.944937>
- Bishop and Bishop (2024). Deep Learning: Foundations and Concepts. *Springer Nature*. <https://doi.org/10.1007/978-3-031-45468-4>
- Elhage et al. (2021). A Mathematical Framework for Transformer Circuits. *Transformer Circuits Thread*. <https://transformer-circuits.pub/2021/framework/index.html>
- Ramsauer et al. (2021). Hopfield networks is all you need. *Int. Conf. Learn. Represent. (ICLR)*. <https://openreview.net/forum?id=tL89RnzIiCd>
- Cormen et al. (2022). Introduction to Algorithms. *MIT Press*.
- Kitaev et al. (2020). Reformer: The Efficient Transformer. *Int. Conf. Learn. Represent. (ICLR)*. <https://openreview.net/forum?id=rkgNKkHtvB>

## Cite this entry

```bibtex
@misc{dictml_attention,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {attention},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/attention.html}
}
```


---

# autoencoder

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-07. [HTML](https://dictionaryofml.org/terms/autoencoder.html) · [typeset PDF](https://dictionaryofml.org/terms/autoencoder.pdf) · [LaTeX](https://dictionaryofml.org/terms/autoencoder.tex) · [Python demo](https://dictionaryofml.org/terms/autoencoder.py) · [notebook](https://dictionaryofml.org/terms/autoencoder.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

An autoencoder is a machine learning method that learns an encoder and a decoder together. The encoder maps the feature vector of a data point to a new feature vector, referred to as code. The decoder reconstructs the original feature vector from the code. The code delivered by the encoder is typically much shorter than the original feature vector, so the code can retain only part of the information in the feature vector. Different autoencoders use different choices for the hypothesis spaces from which the encoder and the decoder are chosen. The special case where both are linear maps learns the same subspace as principal component analysis, and a deep autoencoder uses a deep net on each side.

## Definition

Three hundred points along a curve in the plane (Fig. 3) can each be stored as a single number from which the point is rebuilt almost exactly, and an image can likewise be stored as a short code from which a close copy is rebuilt. An autoencoder learns an encoder $h$ from a hypothesis space $\mathcal{H}$ along with a decoder $h^{*}$ from a second hypothesis space $\mathcal{H}^{*}$ (Goodfellow et al., 2016, Ch. 14). The encoder maps the feature vector ${\mathbf x} \in \mathbb{R}^{d}$ of a data point to a code ${\mathbf z} = h({\mathbf x}) \in
\mathbb{R}^{d'}$. The decoder maps the code back to a reconstruction $\widehat{{\mathbf x}} = h^{*}({\mathbf z})$ of the original feature vector (see Fig. 1). The encoder and the decoder can be learnt via empirical risk minimization with a loss that measures the reconstruction error ${\mathbf x} -
h^{*}(h({\mathbf x}))$, such as the squared error loss (Goodfellow et al., 2016, Sect. 14.1),

\begin{equation}
\tag{1}
\min_{h \in \mathcal{H},\; h^{*} \in \mathcal{H}^{*}}
\frac{1}{m} \sum_{r=1}^{m}
L\Big( {\mathbf x}^{(r)},\,
h^{*}\big( h\big( {\mathbf x}^{(r)} \big) \big) \Big)
\text{,}
\end{equation}

The training set needs no labels: what the decoder has to reproduce is the data point itself, so an autoencoder is a form of self-supervised learning, as it does not require manual labeling.

![Figure 1 of the entry autoencoder](https://dictionaryofml.org/terms/fig/autoencoder-fig1.svg)

*Figure 1: An autoencoder: the encoder $h$ maps a feature vector ${\mathbf x}$ to a code ${\mathbf z}$, and the decoder $h^{*}$ maps the code back to a reconstruction $\widehat{{\mathbf x}}$. Both are learned jointly, by minimizing a quantitative measure of the reconstruction error ${\mathbf x} - h^{*}(h({\mathbf x}))$*

The length $d'$ of the code is typically much smaller than the length $d$ of the original feature vector, in which case the autoencoder is called undercomplete (Goodfellow et al., 2016, Sect. 14.1). The encoder then implements a form of compression that still allows the decoder to reconstruct the original feature vector with a small reconstruction error (Hinton and Salakhutdinov, 2006). The encoder map can afterwards be used as a feature transformation for learning tasks based on the same data points, which makes the autoencoder a feature learning and a dimensionality reduction method (Bengio et al., 2013). The autoencoder is rarely used for the reconstruction itself: the decoder is dropped and the code kept, so that a later learning task learns a hypothesis with few model parameters on the code rather than on the raw features (see transfer learning).

Different autoencoder methods are obtained by different choices for the hypothesis spaces $\mathcal{H}$, $\mathcal{H}^{*}$ and the loss function in Eq. (1). The special case of linear maps for the encoder and the decoder, paired with the squared error loss, learns the same subspace as principal component analysis. At every minimizer of Eq. (1), the composition $h^{*} \circ h$ is the projection onto the span of the first $d'$ principal components (Bishop, 2006, Sect. 12.4.2). The code itself can be the coordinates with respect to any basis of that subspace, not only the orthonormal one principal component analysis delivers. For some applications, nonlinear encoder and decoder maps represented by ANNs are more useful, and a deep autoencoder uses a deep net on each side (Fig. 2). On the curve of Fig. 3, a code of one number and one hidden layer on each side leave an average squared error loss of $0.003$, against $0.457$ for the best linear map with the same code. The gain is not the ANNs as such but the shape of the reconstruction: linear maps with a code of one number can only place the reconstructions on a straight line, and these data points do not lie on one.

![Figure 2 of the entry autoencoder](https://dictionaryofml.org/terms/fig/autoencoder-fig2.svg)

*Figure 2: A deep autoencoder. The encoder $h$ is a deep net of three layers that maps the feature vector ${\mathbf x}$, here $d = 6$ numbers, to a code ${\mathbf z}$ of $d' = 2$ numbers (shaded). The decoder $h^{*}$ rebuilds the feature vector from that code with two further layers. The code is the narrowest layer, so everything the reconstruction uses has to pass through two numbers*

![Figure 3 of the entry autoencoder](https://dictionaryofml.org/terms/fig/autoencoder-fig3.svg)

*Figure 3: Three hundred data points along a curve, compressed to a code of one number and rebuilt; the dots are $20$ of them, every $15$th. Linear maps place every reconstruction on a straight line (dashed line), which no choice of line brings close to these data points; ANNs follow the curve (solid curve), apart from a small loop near $x_{1} = 0.6$ that is a defect of the learned decoder, and leave an average squared error loss $150$ times smaller. Data generated by pythondemos/autoencoder.py*

**See also:** encoder, decoder, [feature learning](https://dictionaryofml.org/terms/featlearn.html), feature transformation, [dimensionality reduction](https://dictionaryofml.org/terms/dimred.html), [principal component analysis](https://dictionaryofml.org/terms/pca.html), [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), squared error loss, [self-supervised learning](https://dictionaryofml.org/terms/selfsupervisedlearning.html), [transfer learning](https://dictionaryofml.org/terms/transferlearning.html), [artificial neural network](https://dictionaryofml.org/terms/ann.html), deep net.

## References

- Goodfellow et al. (2016). Deep Learning. *MIT Press*.
- Hinton and Salakhutdinov (2006). Reducing the dimensionality of data with neural networks. *Science*. <https://doi.org/10.1126/science.1127647>
- Bengio et al. (2013). Representation learning: a review and new perspectives. *IEEE Trans. Pattern Anal. Mach. Intell.*. <https://doi.org/10.1109/TPAMI.2013.50>
- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>

## Cite this entry

```bibtex
@misc{dictml_autoencoder,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {autoencoder},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/autoencoder.html}
}
```


---

# backpropagation

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-10. [HTML](https://dictionaryofml.org/terms/backpropagation.html) · [typeset PDF](https://dictionaryofml.org/terms/backpropagation.pdf) · [LaTeX](https://dictionaryofml.org/terms/backpropagation.tex) · [Python demo](https://dictionaryofml.org/terms/backpropagation.py) · [notebook](https://dictionaryofml.org/terms/backpropagation.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Backpropagation is an algorithm that computes the gradient of the loss with respect to the model parameters of an artificial neural network. The algorithm consists of two consecutive phases, referred to as forward and backward pass. The forward pass evaluates, for the current model parameters, the activations and the resulting prediction errors on a given dataset. The backward pass then applies the chain rule to compute the partial derivatives of the average loss with respect to the model parameters. These partial derivatives constitute a gradient, which can be used in a gradient step to update the model parameters.

## Definition

Consider an artificial neural network with a billion model parameters and its prediction for a new data point. If this prediction is wrong, the model parameters can be updated by a gradient step. That update requires the partial derivatives of the loss on the new data point with respect to all model parameters.

Backpropagation is an algorithm for computing these partial derivatives with as little duplicated computation as possible (Goodfellow et al., 2016, Sect. 6.5.3; Rumelhart et al., 1986). More generally, it computes the gradient $\nabla f({\mathbf w})$ of an objective function $f({\mathbf w})$ with respect to the model parameters ${\mathbf w}$ of the artificial neural network. The model parameters consist of the edge weights and the offset terms at the nodes of the artificial neural network. The objective function is typically the average loss

$$
f({\mathbf w}) = \frac{1}{m}
\sum_{r=1}^{m}
L\left(\left( {\mathbf x}^{(r)},y^{(r)} \right),h^{({\mathbf w})} \right)
\text{,}
$$

incurred on a dataset of $m$ data points $\left( {\mathbf x}^{(1)},y^{(1)} \right), \ldots,
\left( {\mathbf x}^{(m)},y^{(m)} \right)$. Backpropagation computes the partial derivatives of $f({\mathbf w})$ one layer at a time (Fig. 1). Its execution consists of two consecutive phases, the forward pass and the backward pass.

In the forward pass, the artificial neural network is applied to each data point of the dataset: the neurons of each layer evaluate their activations from the activations of the previous layer. The final layer then produces the output, which is the prediction for the label of a data point with features ${\mathbf x}$. The quality of this prediction is measured by a loss function $L\left(\left( {\mathbf x},y \right),h^{({\mathbf w})} \right)$, where $h^{({\mathbf w})}$ is the hypothesis map represented by the artificial neural network with model parameters ${\mathbf w}$. In the backward pass, the partial derivatives of $f({\mathbf w})$ with respect to the edge weights of each layer are computed recursively, from the output back to the input. These partial derivatives form the whole gradient $\nabla f({\mathbf w})$. The backward pass requires about as much computation as the forward pass: its cost grows in proportion to the number of edges of the artificial neural network, one partial derivative, one multiplication, and one addition per edge (Goodfellow et al., 2016, Sect. 6.5.3).

![Figure 1 of the entry backpropagation](https://dictionaryofml.org/terms/fig/backpropagation-fig1.svg)

*Figure 1: Backpropagation through an artificial neural network with one hidden layer. The solid arrows indicate the forward pass: the features $x_{1},x_{2},x_{3}$ reach the neurons $a_{1},a_{2},a_{3}$ through the edge weights ${\mathbf W}^{(1)}$, then the output $\widehat{y} = h^{({\mathbf w})}({\mathbf x})$ through ${\mathbf W}^{(2)}$. The output is a prediction for the label of a data point and incurs the loss $f({\mathbf w})$. The dashed arrow indicates the backward pass, which runs the other way and yields the partial derivatives of $f({\mathbf w})$ with respect to the edge weights of both layers. The model parameters ${\mathbf w}$ collect the edge weights of both layers; the offset terms of the neurons are omitted here*

To illustrate the details of backpropagation, consider a simple artificial neural network: it consists of three layers with one neuron each. Its model parameters are the three edge weights $w^{(1)}, w^{(2)}, w^{(3)}$. The gradient of an average is the average of the gradients of its summands, so a dataset with a single data point already shows the recursion. The objective function $f({\mathbf w})$ is then the loss of that data point. Write $a^{(0)} = x$ for its feature. Layer $\ell = 1, 2, 3$ computes

$$
z^{(\ell)} = w^{(\ell)} a^{(\ell-1)} \text{,} \qquad
a^{(\ell)} = \sigma\big(z^{(\ell)}\big) \text{,}
$$

with the activation function $\sigma$ and no offset terms. The output is $h^{({\mathbf w})}(x) = a^{(3)}$. The edge weight $w^{(3)}$ enters $f({\mathbf w})$ only through $z^{(3)}$, $w^{(2)}$ through $z^{(2)}$ and then $z^{(3)}$, and $w^{(1)}$ through all three.

The chain rule therefore gives

$$
\begin{aligned}
\frac{\partial f}{\partial w^{(3)}}
&= \frac{\partial f}{\partial a^{(3)}}\,
\sigma'\big(z^{(3)}\big)\, a^{(2)} \text{,} \\
\frac{\partial f}{\partial w^{(2)}}
&= \frac{\partial f}{\partial a^{(3)}}\,
\sigma'\big(z^{(3)}\big)\, w^{(3)}\,
\sigma'\big(z^{(2)}\big)\, a^{(1)} \text{,} \\
\frac{\partial f}{\partial w^{(1)}}
&= \frac{\partial f}{\partial a^{(3)}}\,
\sigma'\big(z^{(3)}\big)\, w^{(3)}\,
\sigma'\big(z^{(2)}\big)\, w^{(2)}\,
\sigma'\big(z^{(1)}\big)\, x \text{.}
\end{aligned}
$$

Each line repeats the one above it and appends two factors. Evaluating the three lines separately would recompute the repeated factors twice over. Backpropagation computes them once, from the output backwards. The repeated factors form the partial derivative of $f({\mathbf w})$ with respect to $z^{(\ell)}$. This partial derivative obeys its own recursion,

$$
\frac{\partial f}{\partial z^{(3)}}
= \frac{\partial f}{\partial a^{(3)}}
\sigma'\big(z^{(3)}\big) \text{,} \qquad
\frac{\partial f}{\partial z^{(\ell)}}
= \frac{\partial f}{\partial z^{(\ell+1)}}
w^{(\ell+1)} \sigma'\big(z^{(\ell)}\big)
\quad \text{for } \ell = 2, 1 \text{.}
$$

Each edge weight then costs one multiplication,

$$
\frac{\partial f}{\partial w^{(\ell)}}
= \frac{\partial f}{\partial z^{(\ell)}} a^{(\ell-1)}
\text{.}
$$

The backward pass is this recursion, and the activations $a^{(\ell)}$ it needs are the ones the forward pass already computed.

The partial derivatives delivered by backpropagation constitute the gradient used in a gradient step:

$$
{\mathbf w}^{(t+1)} = {\mathbf w}^{(t)}
- \eta \nabla f({\mathbf w}^{(t)})
\text{, for } t = 0,1,\ldots \text{,}
$$

with step size $\eta$. Each gradient step requires a new run of backpropagation, since the model parameters change after each step. These partial derivatives are not only used for a gradient step. A class activation map explains a prediction of a convolutional neural network by weighting its feature maps with partial derivatives of the output (Selvaraju et al., 2017).

**See also:** [artificial neural network](https://dictionaryofml.org/terms/ann.html), deep net, [gradient](https://dictionaryofml.org/terms/gradient.html), [gradient descent](https://dictionaryofml.org/terms/gd.html), gradient step, layer, edge weight, model parameter, activation function, loss function, neuron.

## References

- Goodfellow et al. (2016). Deep Learning. *MIT Press*.
- Rumelhart et al. (1986). Learning Representations by Back-Propagating Errors. *Nature*. <https://doi.org/10.1038/323533a0>
- Selvaraju et al. (2017). Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. *2017 IEEE Int. Conf. Comput. Vis.*. <https://doi.org/10.1109/ICCV.2017.74>

## Cite this entry

```bibtex
@misc{dictml_backpropagation,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {backpropagation},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/backpropagation.html}
}
```


---

# bagging

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-10-31. [HTML](https://dictionaryofml.org/terms/bagging.html) · [typeset PDF](https://dictionaryofml.org/terms/bagging.pdf) · [LaTeX](https://dictionaryofml.org/terms/bagging.tex) · [Python demo](https://dictionaryofml.org/terms/bagging.py) · [notebook](https://dictionaryofml.org/terms/bagging.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Bagging is an ensemble method that uses randomly sampled subsets of a dataset as training sets for its base learners. Aggregating the predictions delivered by the base learners, by a majority vote in classification or by averaging in regression, yields the prediction of the ensemble method. A random forest is the instance of bagging whose base learners learn decision trees.

## Definition

On twenty-four days of 2020, twelve from winter and twelve from summer, the weather station Krems an der Donau recorded a minimum and a maximum temperature. Predicting the maximum temperature $y$ of a day from its minimum temperature $x$ is a regression problem, and a decision tree solves it by splitting the feature axis at a threshold and predicting the mean label on each side.

Fig. 1 shows what happens when two decision trees are each learned from five of those days chosen at random: they place their threshold at $5.4$ and $9.2$ degrees Celsius, and their predictions differ by up to $22$ degrees Celsius. This spread indicates a lack of stability of the base learner: it is too sensitive to the specific choice of training set, which in turn implies poor generalization.

Bagging is an ensemble technique that suppresses this instability. Bagging executes its base learners in parallel, in contrast to boosting, which runs its base learners sequentially. The name is short for bootstrap aggregating.

![Figure 1 of the entry bagging](https://dictionaryofml.org/terms/fig/bagging-fig1.svg)

*Figure 1: Twenty-four days of 2020 at Krems an der Donau, twelve from winter and twelve from summer, with the minimum temperature of a day as feature and its maximum temperature as label. Two decision trees of depth one, each learned from five of the days chosen at random, are drawn in the style of the days they were trained on: the solid black step curve with the filled circles, the dashed gray step curve with the filled squares. The remaining days are the small gray dots. The two trees place their threshold at different temperatures and predict different levels on either side of it. Data generated by pythondemos/bagging.py*

Bagging uses randomly sampled subsets of a dataset as training sets for its base learners. Given the original training set $\mathcal{D}$, it generates perturbed copies $\widetilde{\mathcal{D}}^{(1)},\,\ldots,\,
\widetilde{\mathcal{D}}^{(M)}$ (typically via bootstrap sampling) and trains one base learner on each, producing hypotheses $\widehat{h}^{(1)},\,\ldots,\,
\widehat{h}^{(M)}$. The final hypothesis aggregates these by majority vote for classification (i.e., the class predicted by the most base learners) or averaging for regression, a task whose label space is $\mathbb{R}$ (see Fig. 2). Algorithm 1 states the method in full. Bagging methods differ in their choice of base learner. A random forest, for example, is the instance of bagging whose base learners learn decision trees.

![Algorithm 1 of the entry bagging](https://dictionaryofml.org/terms/fig/bagging-alg1.svg)

*Algorithm 1: Bagging*

![Figure 2 of the entry bagging](https://dictionaryofml.org/terms/fig/bagging-fig2.svg)

*Figure 2: Example of bagging in which three base learners use perturbations $\widetilde{\mathcal{D}}^{(1)},\,\ldots,\,\widetilde{\mathcal{D}}^{(3)}$ of the original training set $\mathcal{D}$ to learn the hypotheses $\widehat{h}^{(1)},\,\ldots,\,\widehat{h}^{(3)}$. The final hypothesis is obtained by aggregating these individual hypotheses via some aggregation rule $\phi_{\rm agg}$. One aggregation rule for regression problems is the average of the individual predictions*

Bagging reduces the variance of the machine learning method used for the base learners (Hastie et al., 2009, Sect. 8.7): each base learner is trained on a different resample, so their prediction errors are only partly correlated. In an idealized setting of bagging with $M$ base learners that deliver uncorrelated predictions, each with variance $\sigma^{2}$, the averaged prediction has variance $\sigma^{2}/M$. In general the predictions are correlated. For $M$ identically distributed predictions, each with variance $\sigma^{2}$ and with pairwise correlation $\rho$, the averaged prediction has variance

$$
\rho \sigma^{2}
+ \frac{1-\rho}{M}\, \sigma^{2}
\text{.}
$$

The second term vanishes as $M$ grows, while the first does not: the correlation between the base learners sets a lower bound $\rho \sigma^{2}$ that no number of base learners brings down (Hastie et al., 2009, Eq. (15.1)). Fig. 3 shows, in a small numerical experiment, the estimated variance of the average of $M$ bagged decision trees against the value this identity predicts from an estimated $\sigma^{2}$ and $\rho$.

The correlation $\rho$ comes from the shared training set. The predictions of the base learners would be independent only if the training set were fixed and given, but it is itself modeled as a random variable, and every base learner is trained on a resample of the same $m$ data points: a resample holds any one of them with probability about $1 - 1/e \approx 63.2\%$ (Hastie et al., 2009, Eq. (7.55)), two resamples therefore share about $(1-1/e)^{2} \approx 40\%$, and all of them share the same empirical distribution.

Bagging reduces only the variance of a machine learning method, not its bias. For the regression setting, the average and each of its members have the same expectation, so averaging leaves the bias of a member unchanged.

![Figure 3 of the entry bagging](https://dictionaryofml.org/terms/fig/bagging-fig3.svg)

*Figure 3: The variance of the average of $M$ bagged decision trees (filled circles), against the value the identity predicts from a $\sigma^{2}$ and a $\rho$ estimated on separate training sets (dashed curve). The two agree to within $5\%$ at every $M$, and both approach the lower bound $\rho \sigma^{2} = 0.0436$ (dotted line) rather than zero. Data generated by pythondemos/bagging.py*

Averaging also yields stability: the learned hypothesis depends only weakly on any single data point of the training set. Consider any machine learning method with predictions in $[0,1]$ whose base learners are each trained on a share $p$ of the $m$ data points, drawn without replacement, and whose predictions are averaged over all such subsamples. Then, removing a single data point from the training set changes the prediction at a test point by more than $\varepsilon$ for at most a $\delta$-fraction of the removed data points, for every pair $(\varepsilon, \delta)$ with $\delta \varepsilon^{2} \geq
\frac{1}{4(m-1)} \cdot \frac{p}{1-p}$ (Soloff et al., 2024, Thm. 8). This guarantee holds for every training set and requires nothing of the base learner beyond the bounded predictions — the stability is created by the averaging itself.

A random forest applies bagging to decision trees and, in addition, lets each split choose only among a small random subset of the features. The random choice reduces the correlation between the learned decision trees, and a lower correlation means a smaller variance of their average (Breiman, 2001; Hastie et al., 2009, Eq. (15.1)).

**See also:** [ensemble](https://dictionaryofml.org/terms/ensemble.html), [bootstrap](https://dictionaryofml.org/terms/bootstrap.html), [random forest](https://dictionaryofml.org/terms/randomforest.html), [base learner](https://dictionaryofml.org/terms/baselearner.html), [variance](https://dictionaryofml.org/terms/variance.html), [boosting](https://dictionaryofml.org/terms/boosting.html), [decision tree](https://dictionaryofml.org/terms/decisiontree.html), [correlation](https://dictionaryofml.org/terms/correlation.html), stability.

## References

- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Soloff et al. (2024). Bagging Provides Assumption-free Stability. *J. Mach. Learn. Res.*. <https://www.jmlr.org/papers/v25/23-0536.html>
- Breiman (2001). Random Forests. *Machine Learning*. <https://doi.org/10.1023/A:1010933404324>

## Cite this entry

```bibtex
@misc{dictml_bagging,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {bagging},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/bagging.html}
}
```


---

# base learner

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-04. [HTML](https://dictionaryofml.org/terms/baselearner.html) · [typeset PDF](https://dictionaryofml.org/terms/baselearner.pdf) · [LaTeX](https://dictionaryofml.org/terms/baselearner.tex) · [Python demo](https://dictionaryofml.org/terms/baselearner.py) · [notebook](https://dictionaryofml.org/terms/baselearner.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A base learner is the machine learning method that an ensemble method runs several times: an algorithm that reads a training set and returns a learned hypothesis. The ensemble method combines the learned hypotheses of its base learners into a single hypothesis. The main approaches differ in how they build the training sets and how they combine the results. Bagging trains its base learners in parallel on samples drawn by the bootstrap, and boosting trains them in sequence, each on a training set that the earlier results modify. Stacking instead treats their predictions as a feature transformation and learns an aggregation rule on the transformed features.

## Definition

An ensemble that predicts tomorrow's maximum temperature at Krems an der Donau averages the predictions of a linear model, a decision tree, and an artificial neural network, each learned on its own from the same training set. Each of the three is a base learner: a machine learning method used within an ensemble. Formally, it is an algorithm $\mathcal{A}$ that reads a training set $\mathcal{D}^{(\rm train)}$ and produces a learned hypothesis $\widehat{h}=\mathcal{A}(\mathcal{D}^{(\rm train)})$. An ensemble method runs several base learners and combines their learned hypotheses into a more suitable hypothesis.

The main approaches differ in how they construct the training sets and combine the results. Bagging trains base learners in parallel on samples generated by the bootstrap. Boosting trains them sequentially, using earlier results to modify the training set for the next base learner. Stacking uses their predictions as a feature transformation and learns an aggregation rule from the transformed features (Hastie et al., 2009, Ch. 8).

**See also:** [ensemble](https://dictionaryofml.org/terms/ensemble.html), [bagging](https://dictionaryofml.org/terms/bagging.html), stacking, [boosting](https://dictionaryofml.org/terms/boosting.html).

## References

- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>

## Cite this entry

```bibtex
@misc{dictml_baselearner,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {base learner},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/baselearner.html}
}
```


---

# baseline

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-11. [HTML](https://dictionaryofml.org/terms/baseline.html) · [typeset PDF](https://dictionaryofml.org/terms/baseline.pdf) · [LaTeX](https://dictionaryofml.org/terms/baseline.tex) · [Python demo](https://dictionaryofml.org/terms/baseline.py) · [notebook](https://dictionaryofml.org/terms/baseline.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A baseline is a reference level against which the performance of a trained model is compared. The cheapest baseline uses no features at all and predicts the same value for every data point, e.g., the sample mean of numeric label values, or the most frequent label value in classification. Any useful method must beat that level. Baselines can be obtained from human performance, from an existing machine learning method, or from a probabilistic model. Given an accurate probabilistic model of the data generation, the smallest achievable risk is the Bayes risk, incurred by the Bayes estimator of the label given the features. In practice, however, the Bayes estimator is infeasible for at least two reasons. First, the probability distribution of the data generation process might be impossible to determine. Second, computing the Bayes risk might be computationally too expensive.

## Definition

On two days of 2024, a weather station Krems an der Donau (Austria) recorded the same minimum temperature of $11.8\,^{\circ}\mathrm{C}$, but a maximum of $13.2\,^{\circ}\mathrm{C}$ on one day and $29.9\,^{\circ}\mathrm{C}$ on the other. No hypothesis that predicts the maximum daytime temperature from the minimum can be right on both days. Whatever loss a hypothesis incurs on those two days is unavoidable, so a measured loss above zero is not by itself a sign of a poor machine learning method.

More generally, consider a machine learning method that produces a learned hypothesis (or trained model) $\widehat{h} \in \mathcal{H}$. It is evaluated by computing the average loss on a test set (Hastie et al., 2009, Sect. 7.2). The question is whether the learned hypothesis is already close to optimal. If it is, little is gained by collecting more data or by spending more computation to improve it. Answering the question needs a baseline: a reference level for the average loss, either the smallest that any method can achieve or the level that any useful method must beat.

Sometimes a baseline is computed from the dataset itself, as in the scatterplot of Fig. 1 (Hastie et al., 2009, Sect. 2.4). The figure shows weather records of the station Krems an der Donau (Austria). Each of the 366 days of 2024 is a data point whose feature $x$ is the minimum air temperature of the day and whose label $y$ is its maximum air temperature. For the 15 data points whose $x$ is between $11.5\,^{\circ}\mathrm{C}$ and $12.5\,^{\circ}\mathrm{C}$, the corresponding label ranged from $13.2\,^{\circ}\mathrm{C}$ to $33.4\,^{\circ}\mathrm{C}$. Two of these data points even share the same feature value $x = 11.8$, with labels $13.2$ and $29.9$. A hypothesis is a map, so it assigns both days the same prediction. On these two data points it therefore incurs an average squared error loss of at least $(16.7/2)^{2} \approx 69.7$. The same holds wherever data points share a feature value: the spread of their labels determines an average loss that no hypothesis, however learned, can avoid.

The gray strips of Fig. 1 collect data points with very similar feature values. Each strip holds at least four data points and is drawn over the range of labels those data points carry. A strip is therefore narrow where data points are dense along the feature axis and wide where they are sparse, and it is tall wherever those data points carry very different labels. Computing the sample variance of the labels within each strip and averaging the results across all strips yields $18.75$ (in squared degrees Celsius). The resulting value, $18.75$ squared degrees Celsius, is an estimate of the smallest average squared error loss any hypothesis can achieve on these data points (Hastie et al., 2009, Sect. 2.4). The hypothesis learned by linear regression, the solid line of Fig. 1, incurs an average squared error loss of $19.17$, its training error. That is within three percent of the baseline and evidence that little is left to gain.

![Figure 1 of the entry baseline](https://dictionaryofml.org/terms/fig/baseline-fig1.svg)

*Figure 1: Observations collected at the weather station Krems an der Donau during 2024. Each data point is one day, with the minimum temperature as its feature $x$ and the maximum temperature as its label $y$. The shaded gray strips provide the data-derived baseline of $18.75$ squared degrees Celsius. The gray strips cut the feature axis into pieces that each hold at least four data points, each drawn over the range of labels its data points carry; their widths differ because the data points are not spread evenly over the feature axis. The open triangles mark the $15$ data points of one such strip, the one around $x=12$, whose feature values lie in $[11.5,12.5)$, and the double arrow spans the labels those $15$ data points carry. The filled circles are $20$ of the $351$ data points outside that strip, every $18$th of them. The solid line is the hypothesis learned by linear regression. Averaged over all strips, a spread of this kind estimates the smallest achievable average loss for any machine learning method that predicts the maximum temperature solely from the minimum temperature. Data generated by pythondemos/baseline.py*

Baselines can provide lower bounds or upper bounds on the achievable average loss. The strip-wise estimate of Fig. 1 approximates a lower bound, the Bayes risk defined below: no hypothesis has a smaller risk.

An upper bound, i.e., a baseline that any useful method should beat, comes from a trivial hypothesis map that does not depend on the features at all. For numeric labels, such a trivial prediction could be the sample mean of the labels in the training set. In classification, the constant prediction could be the label value of the majority class in the training set. Its accuracy is the share of that class, which on imbalanced data is already high. For the temperature dataset of Fig. 1, the constant prediction by the sample mean incurs an average squared error loss of $84.64$, more than four times the above baseline value $18.75$.

A second source of a baseline is a probabilistic model, which provides a precise notion of a baseline that applies to any machine learning method. In particular, the Bayes risk is the risk of the optimal hypothesis: the Bayes estimator of the label $y$ of a data point, given its features ${\mathbf x}$ (Lehmann and Casella, 1998, Thm. 1.1, p. 228). For the squared error loss, the Bayes risk equals the expectation of the posterior variance, taken over the features (Bishop, 2006, Sect. 1.5.5, Eq. (1.90)). The strip-wise estimate in Fig. 1 approximates the Bayes risk by local averaging (Hastie et al., 2009, Sect. 2.4): the sample variance of the labels within a strip around feature value $x'$ approximates the conditional variance $\sigma^{2}_{y \mid x'}$. The average of the strip-wise sample variances approximates the expectation $\mathbb{E}\{\sigma^{2}_{y \mid x}\}$.

This probabilistic interpretation also motivates the constant baseline above. The constant prediction of the majority class is itself a Bayes estimator for a probabilistic model $\mathbb{P}^{(y)}$ that ignores the features of data points. Under the 0/1 loss, the Bayes estimator assigns the label with the largest posterior probability (Bishop, 2006, Sect. 1.5.1). For a probabilistic model that does not consider features, the posterior distribution equals the prior distribution of the label and, in turn, the Bayes estimator is the most probable label. The Bayes risk is then the probability that the label takes a value that is different from the most likely one.

For a given choice of loss function, the Bayes estimator (if it exists) is fully determined by the probability distribution $\mathbb{P}$ (Lehmann and Casella, 1998, Thm. 1.1, p. 228). However, computing the Bayes estimator and Bayes risk presents two main challenges. First, the probability distribution $\mathbb{P}$ is unknown and must be estimated from observed data (Hastie et al., 2009, Sect. 2.4). Second, even if $\mathbb{P}$ were known, exact probabilistic inference can be computationally infeasible for general probabilistic models, such as multiply connected Bayesian belief networks (Cooper, 1990, Sect. 4).

A widely-used probabilistic model for data points with numeric features and label is the multivariate normal distribution $\left( {\mathbf x},y \right) \sim \mathcal{N}\left({\boldsymbol \mu},{\mathbf C}\right)$. Here, for the squared error loss, the Bayes estimator is the posterior mean $\mu_{y|{\mathbf x}} := \mathbb{E}\{y \mid {\mathbf x}\}$ of the label $y$, given the features ${\mathbf x}$ (Gray, 2009; Lehmann and Casella, 1998, Cor. 1.2, p. 228). The corresponding Bayes risk is the posterior variance $\sigma^{2}_{y|{\mathbf x}}$ (see Fig. 2). It is the baseline a trained hypothesis $\widehat{h}$ is compared against. Fig. 2 shows its prediction $\widehat{h}({\mathbf x})$ beside the posterior mean.

![Figure 2 of the entry baseline](https://dictionaryofml.org/terms/fig/baseline-fig2.svg)

*Figure 2: The probability distribution (curve) of the label $y$ of a data point, given its features ${\mathbf x}$, when $\left( {\mathbf x},y \right)$ follows a multivariate normal distribution. The dashed line marks the posterior mean $\mu_{y|{\mathbf x}}$, the Bayes estimator under the squared error loss; the horizontal segment has length $\sigma_{y|{\mathbf x}}$, the square root of the posterior variance $\sigma^{2}_{y|{\mathbf x}}$ and hence of the Bayes risk; the cross marks the prediction $\widehat{h}({\mathbf x})$ of a trained hypothesis*

For the multivariate normal distribution, this posterior variance does not depend on ${\mathbf x}$ (Bishop, 2006, Sect. 2.3.1, Eq. (2.73)). More specifically, it is the reciprocal of the corresponding diagonal entry ${\mathbf C}^{-1}$, the inverse of the covariance matrix ${\mathbf C}$: $\sigma^{2}_{y|{\mathbf x}} = 1 / [{\mathbf C}^{-1}]_{y,y}$. In practice, the mean vector ${\boldsymbol \mu}$ and the covariance matrix ${\mathbf C}$ can be estimated from a dataset by the sample mean and the sample covariance matrix (Anderson, 2003, Thm. 3.2.1).

The baselines so far are derived from the data or from a probabilistic model. Three further approaches to constructing a baseline use existing methods for determining predictions. First, a baseline might be obtained from human performance, e.g., the misclassification rate of dermatologists who diagnose skin cancer from images (Salinas et al., 2024, Table 1). Second, for LLMs, baselines are often obtained from a benchmark, i.e., a fixed collection of test problems together with an evaluation protocol. As an example, the benchmark GSM8K measures the ability of a large language model to solve grade-school mathematics word problems (Cobbe et al., 2021). A third source is an existing machine learning method that is unsuitable for the intended machine learning application, e.g., because it is computationally too expensive. The test set error obtained from such a method can still serve as a baseline.

**See also:** Bayes risk, Bayes estimator, [test set](https://dictionaryofml.org/terms/testset.html), probabilistic model, risk, [accuracy](https://dictionaryofml.org/terms/accuracy.html), $\bf 0/1$ loss, imbalanced data, [confusion matrix](https://dictionaryofml.org/terms/cm.html), benchmark, multivariate normal distribution.

## References

- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Lehmann and Casella (1998). Theory of Point Estimation. *Springer-Verlag*. <https://doi.org/10.1007/b98854>
- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Cooper (1990). The computational complexity of probabilistic inference using Bayesian belief networks. *Artif. Intell.*. <https://doi.org/10.1016/0004-3702(90)90060-D>
- Gray (2009). Probability, Random Processes, and Ergodic Properties. *Springer Science+Business Media*. <https://doi.org/10.1007/978-1-4419-1090-5>
- Anderson (2003). An Introduction to Multivariate Statistical Analysis. *Wiley*.
- Salinas et al. (2024). A systematic review and meta-analysis of artificial intelligence versus clinicians for skin cancer diagnosis. *npj Digit. Med.*. <https://doi.org/10.1038/s41746-024-01103-x>
- Cobbe et al. (2021). Training Verifiers to Solve Math Word Problems. <https://doi.org/10.48550/arXiv.2110.14168>

## Cite this entry

```bibtex
@misc{dictml_baseline,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {baseline},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/baseline.html}
}
```


---

# binary classification

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/binclass.html) · [typeset PDF](https://dictionaryofml.org/terms/binclass.pdf) · [LaTeX](https://dictionaryofml.org/terms/binclass.tex) · [Python demo](https://dictionaryofml.org/terms/binclass.py) · [notebook](https://dictionaryofml.org/terms/binclass.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Binary classification refers to machine learning applications with a binary label space. The goal of binary classification is to assign each data point to one of two classes, each class represented by one of the two label values. Strictly speaking, a binary classifier is a hypothesis map from the feature space to a binary label space. However, it is customary to construct such a classifier by comparing the value of a real-valued hypothesis map against a threshold. Linear classifiers are obtained from the sign of a linear hypothesis map. A binary classifier partitions the feature space into two decision regions separated by a decision boundary, which is a hyperplane for a linear classifier. The natural loss function is the 0/1 loss, whose average over a dataset is $1$ minus the accuracy. However, this loss function is hard to optimize for training methods, and its average on imbalanced data is dominated by the majority label value, so it does not indicate which of the two kinds of mistake a classifier makes. Practical methods therefore often use other loss functions for model training and validation.

## Definition

A frost warning answers one question: will the temperature in the coming night fall below $0$ degrees Celsius? The answer is yes or no, nothing in between. An email is spam or not. A credit application defaults or not. A biopsy image is malignant or benign. Binary classification is the machine learning problem behind all of these questions: predict, from the features of a data point, which of two values its label, a number $y$, takes.

Binary classification is classification with a label space $\mathcal{Y}$ that holds two elements. Each data point carries one label, and that label takes one of the two values. The two values are usually written (or encoded) as $\mathcal{Y} = \{-1, +1\}$ or $\mathcal{Y} =
\{0, 1\}$. The choice of the two label values (the encoding) is a convention and irrelevant for the basic properties of binary classification methods. However, mathematical expressions about them often assume a particular encoding, such as $\{-1, +1\}$, and need to be modified accordingly if a different encoding, such as $\{0, 1\}$, is used.

Strictly speaking, a binary classifier is a hypothesis map that takes values in a binary label space $\mathcal{Y}$. It is usually built from a real-valued hypothesis map $h: \mathcal{X} \rightarrow \mathbb{R}$ by comparing its value against a threshold. For the label encoding $\mathcal{Y} = \{-1, +1\}$ and the threshold $0$, a data point is then classified by the sign, $\widehat{y} = \operatorname{sign}(h({\mathbf x}))$. The classifier partitions the feature space into two decision regions, one per label value. They are separated by the decision boundary, the set of ${\mathbf x}$ with $h({\mathbf x}) = 0$. The product $y \cdot
h({\mathbf x})$ is the margin of $h$ on the data point $({\mathbf x}, y)$. It is positive exactly when the prediction is right.

For a linear classifier the real-valued hypothesis is $h({\mathbf x}) = {\mathbf w}^{\top} {\mathbf x} + w_{0}$ with model parameters ${\mathbf w}$ and $w_{0}$, so the decision boundary is the hyperplane ${\mathbf w}^{\top}
{\mathbf x} + w_{0} = 0$, a straight line when there are two features (Bishop, 2006, Sect. 4.1.1). The predicted label value is the side of that hyperplane on which ${\mathbf x}$ lies, and the distance of ${\mathbf x}$ from the hyperplane is $|h({\mathbf x})|
/ \Vert {{\mathbf w}} \Vert$. A data point that lies on the side of the other label value is misclassified. Its 0/1 loss is $1$ (Fig. 1).

![Figure 1 of the entry binclass](https://dictionaryofml.org/terms/fig/binclass-fig1.svg)

*Figure 1: Binary classification of data points with two features using a linear classifier. The decision boundary $h({\mathbf x}) = 0$ is a hyperplane, here a straight line, and it cuts the feature space into the two decision regions of the label values $y = +1$ (filled circles, where $h({\mathbf x}) > 0$) and $y = -1$ (open squares, where $h({\mathbf x}) < 0$). The model parameters are scaled so that $\Vert {{\mathbf w}} \Vert = 1$, and the double arrow shows the margin $y \cdot h({\mathbf x})$ of one data point, which is then its distance from the line. One of the $8$ data points is misclassified: it incurs an 0/1 loss of $1$, and the average 0/1 loss is $1/8$*

The shape of the decision regions depends on the model, and so does how closely they follow the training set. Fig. 2 shows three classifiers learned from the same $30$ days of weather recordings. The linear classifier divides the feature space into two half-planes and misclassifies three of the $30$ days. The decision tree compares one feature with a threshold at each of its splits, so its decision regions are unions of axis-parallel rectangles (Hastie et al., 2009, Sect. 9.2.1). Grown until every leaf holds a single label value, it predicts the label value of every day correctly. The artificial neural network has a curve as decision boundary and also predicts every day correctly. On $170$ further days, the linear classifier is the most accurate of the three. The tree and the artificial neural network have adapted their decision regions to single days of the training set, which is overfitting.

![Figure 2 of the entry binclass](https://dictionaryofml.org/terms/fig/binclass-fig2.svg)

*Figure 2: Decision regions of three binary classifiers learned from the same $30$ data points. Each data point represents the weather conditions around a specific day: its features are the minimum and the maximum temperature of the day in degrees Celsius. Filled circles are days followed by frost ($y = +1$), open squares the other days ($y = -1$). The shaded area is the decision region where the classifier predicts $+1$. (a) A linear classifier, whose regions are two half-planes. (b) A decision tree grown until every leaf holds a single label value, whose regions are unions of axis-parallel rectangles. (c) An artificial neural network with one hidden layer of $32$ units, whose decision boundary is a curve. Data generated by pythondemos/binclass.py*

The natural loss function for binary classification is the 0/1 loss. It is $1$ for a wrong label value and $0$ for a right one. Its average over a dataset is therefore the fraction of misclassified data points, i.e., $1$ minus the accuracy. Empirical risk minimization with the 0/1 loss results in computationally hard optimization problems (Shalev-Shwartz and Ben-David, 2014, Ch. 12). Practical methods therefore minimize a convex surrogate of the 0/1 loss (Shalev-Shwartz and Ben-David, 2014, Sect. 12.3). The hinge loss gives the support vector machine, and the logistic loss gives logistic regression. Fig. 3 shows the three loss functions as functions of the margin.

![Figure 3 of the entry binclass](https://dictionaryofml.org/terms/fig/binclass-fig3.svg)

*Figure 3: The three loss functions of binary classification (with label space $\mathcal{Y} = \{-1, +1\}$) as functions of the margin $y \cdot h({\mathbf x})$: the 0/1 loss (solid curve; at margin $0$ it takes the value $1$, marked by a filled circle, and not the value $0$, marked by an open circle), the hinge loss (dashed curve), and the logistic loss (dotted curve). Both surrogates are convex and decreasing functions of the margin, so a more negative margin costs more, while the 0/1 loss only jumps once, at margin $0$*

The average 0/1 loss is not only challenging as an objective for training but can also be misleading. In particular, the average 0/1 loss (and the associated accuracy) does not indicate which of the two mistakes a classifier makes. The two kinds of mistake, a wrong $+1$ and a wrong $-1$, usually have different costs. On imbalanced data, the constant baseline that always returns the majority label value already reaches a high accuracy. Both are reasons to report the confusion matrix, and the precision, recall, and F1 score computed from it, beside the accuracy.

In the frost warning of the opening, the features of a data point are the minimum and the maximum air temperature of one day. Its label is $y = +1$ if the minimum temperature of the following night falls below $0$ degrees Celsius, and $y =
-1$ otherwise. A wrong $-1$ leaves a fruit grower's blossoms unprotected in a frost. A wrong $+1$ only costs one night of covering or heating them. The two mistakes are therefore not interchangeable, and the classifier is chosen so that wrong $-1$ predictions are rare.

**Synonyms:** two-class classification.

**See also:** [classification](https://dictionaryofml.org/terms/classification.html), multi-label classification, classifier, [decision boundary](https://dictionaryofml.org/terms/decisionboundary.html), $\bf 0/1$ loss, [logistic loss](https://dictionaryofml.org/terms/logloss.html), [logistic regression](https://dictionaryofml.org/terms/logreg.html), [accuracy](https://dictionaryofml.org/terms/accuracy.html), [confusion matrix](https://dictionaryofml.org/terms/cm.html), imbalanced data, [baseline](https://dictionaryofml.org/terms/baseline.html).

## References

- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>

## Cite this entry

```bibtex
@misc{dictml_binclass,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {binary classification},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/binclass.html}
}
```


---

# boosting

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/boosting.html) · [typeset PDF](https://dictionaryofml.org/terms/boosting.pdf) · [LaTeX](https://dictionaryofml.org/terms/boosting.tex) · [Python demo](https://dictionaryofml.org/terms/boosting.py) · [notebook](https://dictionaryofml.org/terms/boosting.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Boosting is an ensemble method that sequentially combines simple base learners, also called weak learners. At each iteration, a base learner uses a modified training set constructed from the original training set and the current predictions. Gradient boosting replaces the labels by negative loss partial derivatives, while adaptive boosting retains the labels and changes the weights of the data points. The resulting hypothesis is an additive combination of the learned corrections.

## Definition

The weather station Krems an der Donau records the daily minimum and maximum temperature. Each day is a data point with the minimum temperature as its feature $x$ and the maximum temperature as its label $y$ (Fig. 1). The figure also shows a hypothesis learned by a depth-one decision tree. This tree is too simple to capture the relation between $x$ and $y$. Boosting improves the hypothesis by adding another hypothesis that corrects its prediction errors. The two curves correspond to the first two iterations of this process.

Boosting is an ensemble method that sequentially combines base learners to obtain a more accurate hypothesis (Hastie et al., 2009, Ch. 10). The base learners are typically simple models, such as shallow decision trees, and are also called weak learners.

![Figure 1 of the entry boosting](https://dictionaryofml.org/terms/fig/boosting-fig1.svg)

*Figure 1: Two iterations of boosting on five days of weather measurements at Krems an der Donau. The dashed curve is the hypothesis learned by the first depth-one decision tree. The second decision tree is learned from its residuals. Adding that decision tree gives the solid curve and lowers the average squared prediction error from $31.0$ to $10.9$. Data generated by pythondemos/boosting.py*

In contrast to ensemble methods such as bagging and stacking, boosting trains its base learners sequentially. At iteration $t$, a base learner learns a hypothesis $\widehat{h}^{(t)}$ from a modified training set $\widetilde{\mathcal{D}^{(\rm train)}}^{(t)}$. This training set is constructed from the original training set $\mathcal{D}^{(\rm train)}$ and the predictions of $\widetilde{h}^{(t-1)}$ (Fig. 2).

![Figure 2 of the entry boosting](https://dictionaryofml.org/terms/fig/boosting-fig2.svg)

*Figure 2: Boosting constructs each modified training set $\widetilde{\mathcal{D}^{(\rm train)}}^{(t)}$ from the original training set $\mathcal{D}^{(\rm train)}$ and the current aggregate hypothesis $\widetilde{h}^{(t-1)}$. The dotted arrows show that the original training set and the preceding base learner affect the next modified training set. A base learner learns the next correction from that training set*

Boosting methods start from an initial hypothesis $\widetilde{h}^{(0)}$, which is iteratively improved by the update

\begin{equation}
\widetilde{h}^{(t)} = \widetilde{h}^{(t-1)}+
\eta^{(t)}
\widehat{h}^{(t)}\text{,}\qquad
t=1,\ldots,T\text{.}
\tag{1}
\end{equation}

Here, $\eta^{(t)}$ is a learning rate and $\widehat{h}^{(t)}$ the hypothesis of the $t$-th base learner (see Fig. 3). The additive updates Eq. (1) therefore form a sequence of aggregate hypotheses.

![Figure 3 of the entry boosting](https://dictionaryofml.org/terms/fig/boosting-fig3.svg)

*Figure 3: Boosting constructs a sequence of hypothesis maps. Each new hypothesis map $\widetilde{h}^{(t+1)}$ is obtained from a base learner $\widehat{h}^{(t+1)}$ that aims to correct the prediction errors of the current hypothesis $\widetilde{h}^{(t)}$*

Different boosting methods construct $\widetilde{\mathcal{D}^{(\rm train)}}^{(t)}$ differently. Gradient boosting replaces the label $y$ of each data point $({\mathbf x},y)$ by the negative partial derivative of its loss with respect to the current prediction,

$$
-\left.
\frac{\partial
L\left(({\mathbf x},y),z \right)}{\partial z}
\right|_{z=\widetilde{h}^{(t-1)}({\mathbf x})}.
$$

These values are pseudo-labels, also called negative gradients. This construction applies when the loss function depends on the hypothesis through its prediction. For the squared error loss, the pseudo-labels are proportional to the prediction errors $y-\widetilde{h}^{(t-1)}({\mathbf x})$. Algorithm 1 summarizes a basic form of gradient boosting (Hastie et al., 2009, Algorithm 10.3).

![Algorithm 1 of the entry boosting](https://dictionaryofml.org/terms/fig/boosting-alg1.svg)

*Algorithm 1: Gradient boosting*

Gradient boosting can be viewed as an extension of gradient descent from updates of a finite-dimensional model parameter to updates in a hypothesis space. For the squared error loss and a base learner that uses linear regression, this connection is exact under the orthogonality and normalization condition stated below. At iteration $t$, Algorithm 1 learns a linear map that predicts the residuals $y^{(r)}-\big({\mathbf x}^{(r)}\big)^\top
{\mathbf w}^{(t-1)}$, for $r=1,\ldots,m$. If the feature matrix ${\mathbf X}=\big({\mathbf x}^{(1)},\ldots,
{\mathbf x}^{(m)}\big)^\top$ has full column rank, the model parameter of this map is the unique least-squares solution (Hastie et al., 2009, Sect. 3.2, Eq. (3.6))

$$
\Delta{\mathbf w}^{(t)}=
({\mathbf X}^{\top}{\mathbf X})^{-1}{\mathbf X}^{\top}
\big({\mathbf y}-{\mathbf X}{\mathbf w}^{(t-1)}\big).
$$

If the columns of ${\mathbf X}$ are also orthogonal and normalized such that ${\mathbf X}^{\top}{\mathbf X}=m\mathbf{I}$, then

$$
\Delta{\mathbf w}^{(t)}=
\frac{1}{m}{\mathbf X}^{\top}
\big({\mathbf y}-{\mathbf X}{\mathbf w}^{(t-1)}\big),
$$

which is the negative gradient, at ${\mathbf w}^{(t-1)}$, of the scaled average squared error loss $f({\mathbf w}) := \frac{1}{2m}
\left\lVert {\mathbf y}-{\mathbf X}{\mathbf w} \right\rVert_{2}^{2}$. The factor $1/2$ cancels the factor $2$ that differentiating the square produces. With matching initialization and learning rates, the boosting iterates satisfy $\widetilde{h}^{(t)}=
h^{({\mathbf w}^{(t)})}$ and coincide with the gradient descent iterates.

Adaptive boosting applies boosting to binary classification. Like gradient boosting in Algorithm 1, it uses the additive update Eq. (1). However, it uses a different construction for the training set $\widetilde{\mathcal{D}^{(\rm train)}}^{(t)}$: the prediction errors of $\widetilde{h}^{(t-1)}$ determine weights, with larger weights assigned to misclassified data points.

Algorithm 2 presents adaptive boosting.M1 (Hastie et al., 2009, Algorithm 10.1).

![Algorithm 2 of the entry boosting](https://dictionaryofml.org/terms/fig/boosting-alg2.svg)

*Algorithm 2: Adaptive boosting.M1*

At the start of iteration $t$, the normalized weights satisfy

$$
q^{(r)}=
\frac{\exp\!\left(-y^{(r)}
\widetilde{h}^{(t-1)}
({\mathbf x}^{(r)})\right)}
{\sum_{r'=1}^{m}
\exp\!\left(-y^{(r')}
\widetilde{h}^{(t-1)}
({\mathbf x}^{(r')})\right)}.
$$

Since the normalized sample weights $q^{(r)}$ in the preceding display sum to $1$, the quantity $\epsilon^{(t)}$ defined in Algorithm 2 is the weighted average 0/1 loss of $\widehat{h}^{(t)}$ on the training set used at iteration $t$.

A single iteration of adaptive boosting reduces the average exponential loss, which is the empirical risk

$$
\widehat{L}\big(\widetilde{h}^{(t)}|\mathcal{D}^{(\rm train)}\big)
:= \frac{1}{m}
\sum_{r=1}^{m}
\exp\!\left(-y^{(r)}
\widetilde{h}^{(t)}
({\mathbf x}^{(r)})\right).
$$

Substituting the additive update Eq. (1) into this empirical risk gives

$$
\widehat{L}\big(\widetilde{h}^{(t)}|\mathcal{D}^{(\rm train)}\big)
=\kappa^{(t)}
\widehat{L}\big(\widetilde{h}^{(t-1)}|\mathcal{D}^{(\rm train)}\big),
$$

where

$$
\kappa^{(t)}=(1-\epsilon^{(t)})
e^{-\eta^{(t)}}+\epsilon^{(t)}
e^{\eta^{(t)}}
=2\sqrt{\epsilon^{(t)}(1-\epsilon^{(t)})}.
$$

Thus, if $0<\epsilon^{(t)}<1/2$, then $0<\kappa^{(t)}<1$. For this iteration, scalar multiplication by $\kappa^{(t)}$ is a contractive operator on the average exponential loss. Since the 0/1 loss is bounded by the exponential loss, repeated contractions also reduce an upper bound on the training error (Schapire and Freund, 2012, Thm. 3.1). This argument concerns the empirical loss; it does not assert that the hypotheses themselves form a contractive sequence.

**See also:** [ensemble](https://dictionaryofml.org/terms/ensemble.html), adaptive boosting, gradient boosting, [base learner](https://dictionaryofml.org/terms/baselearner.html), [bagging](https://dictionaryofml.org/terms/bagging.html), contractive operator.

## References

- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Schapire and Freund (2012). Boosting: Foundations and Algorithms. *MIT Press*.

## Cite this entry

```bibtex
@misc{dictml_boosting,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {boosting},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/boosting.html}
}
```


---

# bootstrap

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/bootstrap.html) · [typeset PDF](https://dictionaryofml.org/terms/bootstrap.pdf) · [LaTeX](https://dictionaryofml.org/terms/bootstrap.tex) · [Python demo](https://dictionaryofml.org/terms/bootstrap.py) · [notebook](https://dictionaryofml.org/terms/bootstrap.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

The bootstrap is a stochastic algorithm that generates perturbed variants, referred to as bootstrap samples, of a given dataset. Each bootstrap sample is obtained by randomly selecting data points, with replacement, from the original dataset. The bootstrap samples have the same size as the original dataset. However, they might repeat some of the original data points while omitting others. Several bootstrap samples allow the study of the properties of a machine learning method. For example, a robust machine learning method delivers similar model parameters for two datasets that differ only in the random fluctuations of the data generation. Bootstrap samples can also be used to estimate a confidence interval for a validation error. Another application of the bootstrap is bagging, where the hypotheses learned from the bootstrap samples are aggregated into one.

## Definition

Five days of weather recordings at Krems an der Donau are all the data there is. Each day is a data point, its feature the minimum temperature of the day and its label the maximum temperature of the same day (Fig. 1). This dataset is small: it does not allow the training of a high-dimensional model, and a hypothesis learned from it depends on which five days happened to be recorded. The bootstrap is a stochastic algorithm, stated as Algorithm 1, that generates additional datasets from the recorded one, the so-called bootstrap samples (Efron and Tibshirani, 1993), and so makes that dependence measurable. Fig. 1 shows two bootstrap samples obtained from Algorithm 1. A bootstrap sample can contain a data point of $\mathcal{D}$ several times, so it is a list of data points rather than a set. The bootstrap is based on the independent and identically distributed assumption.

![Algorithm 1 of the entry bootstrap](https://dictionaryofml.org/terms/fig/bootstrap-alg1.svg)

*Algorithm 1: The bootstrap*

![Figure 1 of the entry bootstrap](https://dictionaryofml.org/terms/fig/bootstrap-fig1.svg)

*Figure 1: A dataset of five data points representing daily weather conditions at Krems an der Donau, the $15$th of February, April, June, August, and October $2022$, and below it two bootstrap samples obtained from it by Algorithm 1. In (a) each day is a data point (open circle, numbered). In (b) and (c) the gray open circles repeat the five days, and the filled markers are the five draws of the bootstrap sample: $\widetilde{\mathcal{D}}^{(1)}$ (filled squares) holds the days $2, 3, 3, 4, 5$ and omits day $1$, while $\widetilde{\mathcal{D}}^{(2)}$ (filled triangles) holds the days $1, 2, 2, 4, 5$ and omits day $3$. The second marker of a day drawn twice sits slightly above the first, so that two draws of one day do not hide each other. Weather data: GeoSphere Austria (GeoSphere Austria, 2026). Data generated by pythondemos/bootstrap.py*

Here, a dataset $\mathcal{D} = \big\{ {\mathbf z}^{(1)},
\,\ldots, \,{\mathbf z}^{(m)}\big\}$ is interpreted as the realizations of $m$ independent and identically distributed RVs with a common probability distribution $\mathbb{P}$. That $\mathbb{P}$ is unknown and must be estimated from $\mathcal{D}$ itself. The bootstrap estimates it by the empirical distribution $\mathbb{P}^{(\mathcal{D})}$ of $\mathcal{D}$ (Efron, 1979; Efron and Tibshirani, 1993; Hastie et al., 2009, Sect. 8.2.1). For every event $\mathcal{A}$, the fraction of the data points ${\mathbf z}^{(r)}$ that fall into it approximates its probability by the law of large numbers (Billingsley, 1986, Thm. 6.1; Efron and Tibshirani, 1993, Sect. 4.2),

$$
\frac{1}{m} \big| r: {\mathbf z}^{(r)}
\in \mathcal{A} \big| \approx \mathbb{P}\left(\mathcal{A}\right) \text{.}
$$

The empirical distribution $\mathbb{P}^{(\mathcal{D})}$ is a fully specified probability distribution and can be sampled as often as wanted. As many data points and as many datasets as needed can therefore be generated from the collected dataset. The gain is copies, not information. Each copy keeps the size $m$ of $\mathcal{D}$, because the quantity being imitated is the spread of a dataset of that size drawn from $\mathbb{P}$. Nor does a copy contain new data points: $\mathbb{P}^{(\mathcal{D})}$ places all its mass on the $m$ data points observed, so no draw can produce one that $\mathcal{D}$ does not already contain. A variant, the smoothed bootstrap, draws from a smoothed version of $\mathbb{P}^{(\mathcal{D})}$ in which each data point is replaced by a small bump-shaped probability density function; it produces data points that $\mathcal{D}$ does not contain, which the bootstrap itself never does.

Sampling $m$ data points from $\mathbb{P}^{(\mathcal{D})}$ is the same as drawing $m$ times from $\mathcal{D}$ with replacement, which is what Algorithm 1 does: every draw is made from the whole of $\mathcal{D}$, so a data point can be drawn twice and another not at all (Fig. 2). The number $B$ of bootstrap samples is limited only by the computation they cost.

![Figure 2 of the entry bootstrap](https://dictionaryofml.org/terms/fig/bootstrap-fig2.svg)

*Figure 2: Three bootstrap samples $\widetilde{\mathcal{D}}^{(1)}$, $\widetilde{\mathcal{D}}^{(2)}$, $\widetilde{\mathcal{D}}^{(3)}$ drawn from a dataset $\mathcal{D}$ of $m = 8$ data points (shaded), each by drawing $8$ times with replacement. Every row has the same length as $\mathcal{D}$, repeats some data points and omits others; each of the three happens to contain $5$ of the $8$ distinct data points, against the $0.63 \cdot 8 \approx 5$ that the calculation below predicts*

A data point is left out of a bootstrap sample with probability $(1 - 1/m)^{m}$, so it appears in one with probability $1 - (1 -
1/m)^{m}$, which approaches $1 - 1/e \approx
0.63$ as $m$ grows, since $(1 - 1/m)^{m}$ converges to $1/e$. The number of distinct data points in a bootstrap sample follows from that one probability alone. Write $I_{r}$ for the random variable that takes the value one when ${\mathbf z}^{(r)}$ appears in $\widetilde{\mathcal{D}}^{(b)}$ and zero otherwise. The count of distinct data points is $\sum_{r=1}^{m}
I_{r}$, so its expectation is

$$
{\mathbb{E}} \Big\{ \sum_{r=1}^{m} I_{r}
\Big\} = \sum_{r=1}^{m}
\mathbb{P}\left(I_{r} = 1\right) = m \big( 1 -
(1 - 1/m)^{m} \big) \approx
0.63\,m \text{.}
$$

The $I_{r}$ are not independent: a day drawn twice forces another to be left out. The expectation of a sum is the sum of the expectations nonetheless.

A bootstrap sample therefore holds about $0.63\,m$ distinct data points and overlaps $\mathcal{D}$ heavily. That overlap is what separates the bootstrap from k-fold cross-validation, which partitions $\mathcal{D}$ into blocks that do not overlap at all. Used naively to estimate the prediction error of a learned hypothesis, the bootstrap therefore evaluates it on data points that entered its own training, and the estimate comes out optimistic (Hastie et al., 2009, Sect. 7.11).

Machine learning training methods can be applied separately to each bootstrap sample, giving the learned hypotheses $\widehat{h}^{(1)}, \,\ldots,
\,\widehat{h}^{(B)}$. A quantity computed from $\widetilde{\mathcal{D}}^{(b)}$ is a bootstrap replication (Efron and Tibshirani, 1993), such as the hypothesis $\widehat{h}^{(b)}$ learned by empirical risk minimization from $\widetilde{\mathcal{D}}^{(b)}$ or its prediction $\widehat{h}^{(b)}({\mathbf x})$ for a fixed data point. The spread of the $B$ bootstrap replications estimates how much the output of the machine learning method changes due to random fluctuations in the data generation (Hastie et al., 2009, Sect. 8.2.1).

Fig. 3 shows the bootstrap for learning a linear hypothesis from weather data recorded at Krems an der Donau during $2024$, $m = 366$ days. Each dot is one data point, its feature the minimum temperature and its label the maximum temperature of the day; the figure draws $20$ of them. The thick line is the hypothesis learned from all $366$ by empirical risk minimization with the squared error loss. The thin lines show bootstrap replications $\widehat{h}^{(b)}$, $b = 1, \,\ldots, \,B$, learned by the same empirical risk minimization problem from bootstrap samples. Their spread is narrow here, and the $95\%$ confidence interval for the slope is $[1.05, 1.15]$.

![Figure 3 of the entry bootstrap](https://dictionaryofml.org/terms/fig/bootstrap-fig3.svg)

*Figure 3: Daily minimum and maximum temperature at Krems an der Donau in $2024$. The dots are $20$ of the $m = 366$ data points, every $19$th day of the year. The thick line is the hypothesis $\widehat{h}$ learned from all $m$ data points, the thin lines are the bootstrap replications $\widehat{h}^{(b)}$ learned from the first $25$ of $B = 500$ bootstrap samples, and the spread of all $500$ gives the confidence interval for the slope. Weather data: GeoSphere Austria (GeoSphere Austria, 2026). Data generated by pythondemos/bootstrap.py*

While basic validation gives a point estimate for the risk of a hypothesis, i.e., the validation error, the bootstrap adds a confidence interval to this point estimate. A hypothesis $\widehat{h}$ is learned once from the training set $\mathcal{D}^{(\rm train)}$, and the validation set $\mathcal{D}^{(\rm val)}$ is resampled $B$ times with replacement, each bootstrap sample of the size of $\mathcal{D}^{(\rm val)}$. Evaluating the same $\widehat{h}$ on each of them gives $B$ estimates of the risk. The sample mean of these estimates is the risk estimate, and the square root of their sample variance is its bootstrap standard error (Efron and Tibshirani, 1993, Sect. 6.2). The estimate plus or minus a standard normal quantile times this standard error is a confidence interval (Efron and Tibshirani, 1993, Sect. 13.2).

Bagging is the bootstrap put to a different use: instead of measuring the spread of $\widehat{h}^{(1)}, \,\ldots,
\,\widehat{h}^{(B)}$, it averages them into one hypothesis, which is how a random forest is built from decision trees.

**See also:** empirical distribution, [independent and identically distributed](https://dictionaryofml.org/terms/iid.html), [random variable](https://dictionaryofml.org/terms/rv.html), [probability distribution](https://dictionaryofml.org/terms/probdist.html), [variance](https://dictionaryofml.org/terms/variance.html), [$k$-fold cross-validation](https://dictionaryofml.org/terms/kfoldcv.html), [bagging](https://dictionaryofml.org/terms/bagging.html), confidence interval, validation, [validation set](https://dictionaryofml.org/terms/valset.html), [validation error](https://dictionaryofml.org/terms/valerr.html).

## References

- GeoSphere Austria (2026). GeoSphere Austria climate station archive. <https://dataset.api.hub.geosphere.at/v1/station/historical/klima-v2-1d>
- Efron and Tibshirani (1993). An Introduction to the Bootstrap. *Chapman & Hall*.
- Efron (1979). Bootstrap Methods: Another Look at the Jackknife. *Ann. Statist.*. <https://doi.org/10.1214/aos/1176344552>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Billingsley (1986). Probability and Measure. *Wiley*.

## Cite this entry

```bibtex
@misc{dictml_bootstrap,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {bootstrap},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/bootstrap.html}
}
```


---

# concept activation vector (CAV)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-07. [HTML](https://dictionaryofml.org/terms/cav.html) · [typeset PDF](https://dictionaryofml.org/terms/cav.pdf) · [LaTeX](https://dictionaryofml.org/terms/cav.tex) · [Python demo](https://dictionaryofml.org/terms/cav.py) · [notebook](https://dictionaryofml.org/terms/cav.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A concept activation vector (CAV) is a direction in the activations of one hidden layer of a deep net that stands for a concept named by a user. The user supplies data points that carry the concept and data points that do not, and a linear classifier is trained to tell the two apart from their activations. The CAV is the direction perpendicular to the hyperplane that separates them, pointing to the concept side. Moving the activations of a data point a little along the CAV changes the score the deep net assigns to a label; the rate of that change is the sensitivity of the prediction to the concept. The fraction of data points with a positive sensitivity is the TCAV score. That score belongs to one layer and one label; it aggregates the per-data point sensitivities and thereby describes a whole class of data points rather than a single prediction.

## Definition

A deep net predicts from five consecutive days of weather at a station whether it rains on the sixth day. Whether it uses the notion of a heat wave, five hot days in a row, to arrive at that prediction cannot be decided by inspecting a single neuron. A heat wave is not a feature the network was built with, since its features are the daily temperatures and rain amounts. Nor need it be represented by any one neuron (Kim et al., 2018, Sect. 2.3). A concept is a property that a user can name and exemplify but that the network was not built to compute, such as a heat wave. The concept is specified by examples instead: five-day windows that form a heat wave, the positive examples, and windows that do not, the negative examples (Kim et al., 2018, Sect. 3.1). A CAV turns such a concept into a direction in the activations of one hidden layer, the direction along which the positive examples lie apart from the negative ones.

Consider such a deep net, with several hidden layers, trained to predict the label of a data point from its feature vector; the trained network is the learned hypothesis $\widehat{h}$. Fix one hidden layer and write ${\mathbf z} = f({\mathbf x})$ for its activations, with $f$ the function computed by the layers up to that point. The deep net is applied to the positive and to the negative examples of a concept $\mathcal{C}$. A binary linear classifier $g({\mathbf z})$, whose features are those activations, is trained to tell the two apart. Its decision boundary is a hyperplane; the CAV for $\mathcal{C}$ is the normal vector $\widehat{{\mathbf w}}$ of that hyperplane (Kim et al., 2018, Sect. 3.2).

A hyperplane has two normal vectors, one pointing to each side. The CAV is the one pointing towards the side that $g({\mathbf z})$ assigns to the concept class. Training $g({\mathbf z})$ with the positive examples as the positive class delivers this orientation, and when the two sets of activations are linearly separable, it is the side of the positive examples. In the terminology of mechanistic interpretability, this direction defines a feature of the trained deep net, the projection of the activations onto it: a human-interpretable concept represented by a direction in the space of activations.

Fig. 1 draws the CAV of the heat-wave concept for a deep net with two hidden layers of 16 neurons, trained on the daily weather at Krems an der Donau of the training years 2000–2018 and tested on 2019–2024. A data point is a window of five days, its feature vector holds the daily maximum and minimum temperature and the rain amount of each day, and its label says whether at least 1 mm of rain fell on the sixth day. The concept is specified by 40 heat waves, windows with a daily maximum of at least $30\,^{\circ}$C on all five days, and 40 windows that are not, drawn from the windows of the record. The plane drawn passes through the mean activation of the first hidden layer and is spanned by the CAV and the direction orthogonal to it along which the activations vary most. The concept hyperplane therefore meets the plane in a vertical line, and the CAV is its horizontal axis.

Two classification problems appear in the figure. The first is the concept classification that $g({\mathbf z})$ solves on the activations. The second is the task the deep net is trained for, whose decision boundary on the plane is the solid line: rain is predicted to its right, where every heat wave lies. The CAV depends on the examples that specify the concept: training $g({\mathbf z})$ on all 9127 windows of the record instead turns it by 26 degrees.

![Figure 1 of the entry cav](https://dictionaryofml.org/terms/fig/cav-fig1.svg)

*Figure 1: The heat-wave CAV of a deep net that predicts rain at Krems an der Donau from the five preceding days, drawn in the plane through the mean activation of its first hidden layer spanned by the CAV (horizontal axis) and the direction orthogonal to it along which the activations vary most. Gray dots are $20$ of the $400$ five-day windows of the test years drawn at random. Filled circles are $10$ of the $40$ windows supplied as heat-wave examples and open circles $10$ of the $40$ supplied as non-heat-wave examples, every fourth of each; the dashed line is the decision boundary of the linear classifier trained on all $80$, and the CAV is its normal vector. The solid line is the decision boundary of the deep net on this plane, with rain predicted to its right. Data generated by pythondemos/cav.py*

Like any explanation, a CAV must be understandable and faithful (see explanation). It is understandable by construction: the user supplies the examples that define the concept. Faithfulness is a separate question, because the concept is a direction in the activations whether or not the deep net responds to it. Testing with CAVs (TCAV) decides it.

Write the trained deep net as a concatenation of two functions, the activations ${\mathbf z} = f({\mathbf x})$ of the chosen hidden layer and a score $s({\mathbf z})$ that the remaining layers assign to one label. The conceptual sensitivity of a data point is the directional derivative of that score along the unit CAV ${\mathbf v} := \widehat{{\mathbf w}} / \Vert {\widehat{{\mathbf w}}} \Vert$, which for differentiable $s$ equals the inner product of the gradient of $s$ with ${\mathbf v}$ (Rudin, 1976, Example 9.18),

\begin{equation}
\tag{1}
S({\mathbf x}) := \langle \nabla s\big(f({\mathbf x})\big), {\mathbf v} \rangle
\text{.}
\end{equation}

It measures how much the score for that label moves when the activations move a little towards the concept (Kim et al., 2018, Sect. 3.3). The orientation convention gives the sign its meaning: $S({\mathbf x}) > 0$ says the score rises towards the concept, and the opposite normal vector would flip every sensitivity.

The sensitivity Eq. (1) delivers one number per data point. TCAV collects these numbers over one label: the TCAV score is the fraction of the data points carrying that label for which $S({\mathbf x}) > 0$ (Kim et al., 2018, Sect. 3.4). A random set of non-concept data points also yields a CAV, so a single such fraction is not evidence alone. A random concept is a set of data points drawn at random and labeled as if they carried a concept. It is as likely to raise the score as to lower it, so its TCAV score, the fraction of positive sensitivities, is one half on average. The CAV is therefore refitted against many fresh draws of non-concept data points, and the resulting TCAV scores are compared with those of random concepts: a concept the deep net responds to scores consistently away from one half, a random one does not (Kim et al., 2018, Sect. 3.5).

The explanation delivered is a single number attached to a concept, a label and a layer. Unlike the sensitivity $S({\mathbf x})$, which is defined for a single data point, it describes a whole class of data points rather than a single prediction (Kim et al., 2018, Sect. 2.1).

For the deep net of Fig. 1, moving along the heat-wave CAV raises the rain score at every test window followed by rain. A cold wave is five days with a minimum of at most $-5\,^{\circ}$C, and moving along its CAV lowers that score at every one of those windows. The record agrees: rain follows 30% of the heat waves and 16% of the cold waves, against 23% of all windows.

Training the linear classifier $g({\mathbf z})$ and evaluating the sensitivity $S({\mathbf x})$ both read quantities inside the deep net: the activations $f({\mathbf x})$ of the chosen layer, and the gradient $\nabla s({\mathbf z})$ of the score with respect to them. A CAV is therefore unavailable for a system that returns predictions for submitted feature vectors and nothing else, such as a deep net served behind an interface or supplied by a vendor. An explanation produced by local interpretable model-agnostic explanations, by contrast, needs predictions and nothing else: its local approximation is fitted to predictions obtained by querying the learned hypothesis $\widehat{h}$ (Ribeiro et al., 2016, Sect. 3). A second consequence of reading the internals is that a CAV is not a property of $\widehat{h}$ alone: the split into $f$ and $s$ is a choice of layer, and two deep nets that deliver the same $\widehat{h}$ through different layers can have different CAVs.

Both functions a CAV involves are defined on the space of activations: $g({\mathbf z})$ approximates concept membership there, and the sensitivity $S({\mathbf x})$ is a directional derivative of the score $s$ on that same space. A CAV therefore concerns the interpretability of the deep net rather than its explainability; mechanistic interpretability studies such directions.

**See also:** deep net, [feature](https://dictionaryofml.org/terms/feature.html), [linear classifier](https://dictionaryofml.org/terms/linclass.html), [trustworthy artificial intelligence](https://dictionaryofml.org/terms/trustAI.html), [interpretability](https://dictionaryofml.org/terms/interpretability.html), mechanistic interpretability, [transparency](https://dictionaryofml.org/terms/transparency.html), [explanation](https://dictionaryofml.org/terms/explanation.html), [explainability](https://dictionaryofml.org/terms/explainability.html), neuron, [decision boundary](https://dictionaryofml.org/terms/decisionboundary.html), hyperplane, local interpretable model-agnostic explanations.

## References

- Kim et al. (2018). Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). *Proc. 35th Int. Conf. Mach. Learn.*. <https://proceedings.mlr.press/v80/kim18d.html>
- Rudin (1976). Principles of Mathematical Analysis. *McGraw-Hill*.
- Ribeiro et al. (2016). ``Why Should I Trust You?'': Explaining the Predictions of Any Classifier. *Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining*. <https://doi.org/10.1145/2939672.2939778>

## Cite this entry

```bibtex
@misc{dictml_cav,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {concept activation vector (CAV)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/cav.html}
}
```


---

# classification

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/classification.html) · [typeset PDF](https://dictionaryofml.org/terms/classification.pdf) · [LaTeX](https://dictionaryofml.org/terms/classification.tex) · [Python demo](https://dictionaryofml.org/terms/classification.py) · [notebook](https://dictionaryofml.org/terms/classification.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Classification is the task of predicting a discrete-valued label for a given data point, based solely on its features. The label belongs to a finite label space: binary classification uses two label values, and multi-class classification more than two. A hypothesis whose predictions take values in the finite label space is a classifier; it partitions the feature space into decision regions separated by the decision boundary. A widely used construction compares a real-valued hypothesis against a threshold to obtain the predicted label. A natural quality measure is the 0/1 loss; since it is neither convex nor differentiable, training methods minimize surrogates such as the logistic loss or the hinge loss, and the learned classifier is judged by the accuracy and the confusion matrix on a test set. When the label space is continuous, the task is regression.

## Definition

A weather station records the minimum and the maximum air temperature of each day. From these two numbers, it must be decided whether the following day brings frost, that is, whether its minimum temperature falls below $0\,^{\circ}\mathrm{C}$. A library must file each incoming text into one of the subject categories *mathematics*, *literature*, or *engineering*. Both are classification problems: the task of predicting a discrete-valued label $y$ of a given data point, based solely on its feature vector ${\mathbf x}$, the vector of its features (Bishop, 2006; Duda et al., 2001; Hastie et al., 2009). The label takes values in a finite label space $\mathcal{Y}$ and names the category of the data point. Binary classification uses a label space with two label values, such as $\mathcal{Y} = \{-1, 1\}$ with $y = 1$ marking a frost day; multi-class classification uses more than two label values, such as $\mathcal{Y} = \{\text{mathematics},
\text{literature}, \text{engineering}\}$ for the texts.

When the label space is continuous instead, the task is regression: predicting tomorrow's minimum temperature is regression, predicting whether tomorrow brings frost is classification. Classification is also distinct from clustering: as a form of supervised learning, classification learns from data points whose labels are known, while clustering groups data points without any given labels.

A hypothesis whose predictions take values in a finite label space is referred to as a classifier. A classifier partitions the feature space $\mathcal{X}$ into decision regions, one region per label value, separated by the decision boundary. All feature vectors in the same decision region obtain the same predicted label, so the decision regions determine the classifier completely (Fig. 1).

![Figure 1 of the entry classification](https://dictionaryofml.org/terms/fig/classification-fig1.svg)

*Figure 1: Binary classification with two features $x_{1}, x_{2} \in \mathbb{R}$. The filled circles are data points with $y=-1$, and the open squares are data points with $y=1$. The solid line is the decision boundary; it divides $\mathcal{X}=\mathbb{R}^{2}$ into an unshaded decision region with $\widehat{y}=-1$ and a shaded region with $\widehat{y}=1$. One open square lies in the unshaded region, where the prediction is $\widehat{y}=-1$, so that data point is misclassified. Data generated by pythondemos/classification.py*

A widely used construction of a classifier proceeds in two steps. First, a real-valued hypothesis $h:
\mathcal{X} \rightarrow \mathbb{R}$ quantifies the confidence in one particular label value. Second, the confidence $h({\mathbf x})$ is compared against a threshold to obtain the prediction $\widehat{y} \in \mathcal{Y}$. Logistic regression, for the label space $\mathcal{Y} = \{-1, 1\}$, uses a linear map $h({\mathbf x}) =
{\mathbf w}^{\top} {\mathbf x}$ as the confidence in the label value $1$. This confidence is compared against the threshold $0$,

$$
\widehat{y} := \begin{cases} 1 & \text{if }
h({\mathbf x}) \geq 0\text{,} \\
-1 & \text{otherwise.} \end{cases}
$$

The real-valued hypothesis $h$ is not itself the classifier: the classifier is the thresholded map ${\mathbf x} \mapsto \widehat{y}$, and calling $h$ the classifier is a slight abuse of language. The two decision regions are then the halfspaces on either side of the hyperplane ${\mathbf w}^{\top} {\mathbf x} = 0$, as illustrated in Fig. 1.

A natural quality measure for a classifier is the 0/1 loss: its average over a dataset is the fraction of misclassified data points, and one minus that fraction is the accuracy. Even the best classifier cannot always achieve zero error: when data points with the same feature vector carry different labels, some misclassifications are unavoidable, and the smallest achievable risk is the Bayes risk (Hastie et al., 2009, Sect. 2.4, Eq. (2.23)).

The 0/1 loss is rarely used as the objective function for training: it takes only the values $0$ and $1$, so it is piecewise constant, neither convex nor differentiable as a function of the model parameters, and its minimization is computationally hard (Shalev-Shwartz and Ben-David, 2014, Sect. 12.3). Practical methods therefore minimize the average of a surrogate loss over the training set: a loss function that approximates the 0/1 loss while having more convenient properties, such as convexity or differentiability. Logistic regression uses the logistic loss, which is convex and differentiable (Bishop, 2006, Sect. 4.3.3; Shalev-Shwartz and Ben-David, 2014, Sect. 9.3); the support vector machine uses the hinge loss, which is convex but not differentiable (Boser et al., 1992; Cortes and Vapnik, 1995; Shalev-Shwartz and Ben-David, 2014, Sect. 12.3). Fig. 2 compares the three as functions of the margin $y \cdot h({\mathbf x})$. The learned classifier is then judged by the accuracy and the confusion matrix on a test set.

![Figure 2 of the entry classification](https://dictionaryofml.org/terms/fig/classification-fig2.svg)

*Figure 2: The 0/1 loss (solid line), logistic loss (dashed curve), and hinge loss (dotted line), drawn as functions of the margin $y \cdot h({\mathbf x})$ for $\mathcal{Y} = \{-1, 1\}$. The 0/1 loss is $1$ for a nonpositive margin and $0$ for a positive margin. The logistic loss is convex and differentiable. The hinge loss is convex, but its corner at margin $1$ shows that it is not differentiable there*

**See also:** [label](https://dictionaryofml.org/terms/label.html), label space, classifier, [binary classification](https://dictionaryofml.org/terms/binclass.html), [decision boundary](https://dictionaryofml.org/terms/decisionboundary.html), [decision region](https://dictionaryofml.org/terms/decisionregion.html), $\bf 0/1$ loss, [logistic loss](https://dictionaryofml.org/terms/logloss.html), hinge loss, [logistic regression](https://dictionaryofml.org/terms/logreg.html), [support vector machine](https://dictionaryofml.org/terms/svm.html), [accuracy](https://dictionaryofml.org/terms/accuracy.html), [confusion matrix](https://dictionaryofml.org/terms/cm.html), regression, [supervised learning](https://dictionaryofml.org/terms/supervisedlearning.html), [clustering](https://dictionaryofml.org/terms/clustering.html), Bayes risk.

## References

- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Duda et al. (2001). Pattern Classification. *Wiley*.
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Boser et al. (1992). A Training Algorithm for Optimal Margin Classifiers. *Proc. 5th Annual Workshop on Computational Learning Theory (COLT)*. <https://doi.org/10.1145/130385.130401>
- Cortes and Vapnik (1995). Support-vector networks. *Machine learning*. <https://doi.org/10.1007/BF00994018>

## Cite this entry

```bibtex
@misc{dictml_classification,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {classification},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/classification.html}
}
```


---

# clustering

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/clustering.html) · [typeset PDF](https://dictionaryofml.org/terms/clustering.pdf) · [LaTeX](https://dictionaryofml.org/terms/clustering.tex) · [Python demo](https://dictionaryofml.org/terms/clustering.py) · [notebook](https://dictionaryofml.org/terms/clustering.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Clustering decomposes a dataset into subsets, called clusters, such that data points within the same cluster are more similar to each other than to data points in other clusters. Clustering methods differ in the measure of similarity between data points, in the representation of a cluster, and in whether each data point is assigned to exactly one cluster (hard clustering) or to several with varying degrees of belonging (soft clustering). k-means represents a cluster by a cluster centroid and a Gaussian mixture model by a multivariate normal distribution, so their clusters are convex regions and ellipsoids. Graph clustering represents the data points as the nodes of a similarity graph and uses its weighted edges to determine the clusters.

## Definition

A photograph taken in the Wachau shows sky, clouds, forested slopes, sunlit vineyards and rock (Fig. 1(a)). Cutting it into patches of $5 \times 5$ pixels gives $12288$ data points, each characterized by a feature vector ${\mathbf x} \in \mathbb{R}^{3}$ that holds the mean red, green and blue values of the patch. Clustering is the task of decomposing such a dataset into clusters, such that data points within the same cluster are more similar to each other than to data points in other clusters (Bishop, 2006, Sect. 9.1; Hastie et al., 2009, Sect. 14.3). Painting each patch in the mean color of its cluster divides the photograph into regions, an image segmentation. Fig. 1(b)–(f) shows the segmentations delivered by five clustering methods, k-means, a Gaussian mixture model, density-based spatial clustering of applications with noise, spectral clustering and convex clustering. The number $k$ of clusters is either an input, as in k-means, or determined by the method from its other hyperparameters, as in density-based spatial clustering of applications with noise.

![Figure 1 of the entry clustering](https://dictionaryofml.org/terms/fig/clustering-fig1.svg)

*Figure 1: Image segmentation by clustering. (a) A photograph taken in the Wachau, cut into $96 \times 128$ patches of $5 \times 5$ pixels; the feature vector of a patch holds its mean red, green and blue values. (b)–(f) Each patch painted in the mean color of its cluster, with white lines between clusters: (b) k-means and (c) a Gaussian mixture model, both with $k = 5$; (d) density-based spatial clustering of applications with noise with $\varepsilon = 0.03$ and $m^{*} = 32$, noise patches in black; (e) spectral clustering with $k = 5$ and (f) convex clustering with $\alpha \approx 18$, both using, for each patch, the ten patches of most similar color as its neighbors. Data generated by pythondemos/clustering.py*

Clustering methods differ in three design choices (Jain et al., 1999):

- the measure of similarity between data points (Hastie et al., 2009, Sect. 14.3), e.g., the Euclidean distance between their feature vectors, which makes the feature space a metric space, or the edge weights of a graph whose nodes are the data points;
- the representation of a cluster, e.g., a cluster centroid, a multivariate normal distribution, a set of data points connected by short hops, or a well-connected subset of the nodes of a graph; and
- whether each data point is assigned to exactly one cluster (hard clustering) or to several with varying degrees of belonging (soft clustering).

Fig. 2 illustrates three families of clustering methods: hard clustering, such as k-means; soft clustering, such as a Gaussian mixture model; and graph clustering, where the data points are the nodes of a graph and a cluster is a densely connected subset of nodes.

![Figure 2 of the entry clustering](https://dictionaryofml.org/terms/fig/clustering-fig2.svg)

*Figure 2: Three families of clustering methods, sketched for data points with feature vectors in $\mathbb{R}^{2}$. (a) Hard clustering as in k-means: each data point belongs to exactly one cluster, the one with the nearest cluster centroid (crosses). (b) Soft clustering as in a Gaussian mixture model: two overlapping ellipses (dashed and dotted ellipses), crosses at their means and thin double arrows along their principal directions; a data point in the overlap belongs to both clusters with a degree of belonging between $0$ and $1$. (c) Graph clustering: the data points are the nodes of a graph whose edges connect similar data points, and a cluster is a densely connected subset of nodes with few edges (dashed edges) to the rest*

k-means represents a cluster by a point of the feature space, referred to as cluster centroid. Each data point is assigned to the cluster whose cluster centroid is nearest in Euclidean distance. The cluster centroids are chosen to minimize the average squared Euclidean distance between a data point and its nearest cluster centroid, which Lloyd's algorithm approaches by alternating assignment and averaging (Lloyd, 1982): each step lowers the average, and the iterates converge to a local minimum, not necessarily the global one (Bishop, 2006, Sect. 9.1).

On the photograph, the five cluster centroids of k-means with $k = 5$ are five representative colors, and each cluster consists of the patches whose color is nearest to one of them: sky, cloud, shaded forest, sunlit slopes and rock (Fig. 1(b)). Painting each patch in the color of its cluster centroid also compresses the photograph to five colors. A clustering learned from one photograph is judged by how it assigns new patches: in k-means, a new data point joins the cluster of the nearest cluster centroid, so the cluster centroids are the learned hypothesis.

A Gaussian mixture model represents each cluster by a multivariate normal distribution: a cluster index is drawn first, then the feature vector from that cluster's multivariate normal distribution. The degree of belonging of a data point to a cluster is the posterior distribution of its cluster index, and the expectation–maximization algorithm learns the model parameters by maximum likelihood (Bishop, 2006, Sect. 9.2). The result is a soft clustering whose clusters are ellipsoids, the level sets of a multivariate normal distribution (Bishop, 2006, Sect. 2.3); hard clustering is the extreme case in which every degree of belonging is $0$ or $1$, reached when all clusters share the covariance matrix $\epsilon \mathbf{I}$ and $\epsilon \to 0$ (Bishop, 2006, Sect. 9.3.2). On the photograph, each surface type becomes a multivariate normal distribution over colors, and the degree of belonging of a patch to the sky cluster is the probability that its color was drawn from the multivariate normal distribution of the sky. Painting each patch by the cluster with the largest degree of belonging gives Fig. 1(c); one in eight patches, mostly along the horizon and the cloud edges, has no degree of belonging above $0.8$.

Density-based spatial clustering of applications with noise defines a cluster through two hyperparameters, a distance $\varepsilon$ and a minimum size $m^{*}$ of a cluster (Ester et al., 1996). A data point is a core point if at least $m^{*}$ data points lie within Euclidean distance $\varepsilon$ of it. A cluster is a maximal set of data points that are connected through a chain of core points at most $\varepsilon$ apart, and data points that are neither core points nor within $\varepsilon$ of one are noise. A cluster can therefore have any shape that a chain of nearby data points can trace, and the number of clusters is an output, not an input. On the photograph, $\varepsilon$ is a distance between colors. With the two hyperparameters of Fig. 1(d), the blue of the sky and the white of the clouds are joined by a chain of intermediate colors, so sky and cloud form one cluster and the land another. The patches along the horizon, whose mixed colors are rare, are noise (Fig. 1(d)).

Graph clustering methods represent the dataset as the nodes $r = 1, \ldots, m$ of a graph whose edge weights express the similarity between data points, e.g., the k-nearest-neighbor graph that connects each data point to its nearest neighbors (Luxburg, 2007, Sect. 2). The clusters are determined by the connectivity of the nodes: a cluster is a subset of nodes whose total edge weight to the rest of the graph, divided by the number of nodes in the subset, is small. The sum of this ratio over the clusters is the ratio cut that spectral clustering minimizes approximately (Luxburg, 2007, Sect. 5).

Spectral clustering computes the clusters from eigenvectors of the Laplacian matrix of the graph (Luxburg, 2007; Ng et al., 2001) and flow-based clustering from network flows between selected nodes (SarcheshmehPour et al., 2021). There are also graph clustering methods that incorporate node-wise feature vectors ${\mathbf x}^{(r)}$, $r = 1, \ldots, m$: convex clustering combines the connectivity of the nodes with the node-wise feature vectors. Graph clustering methods suit data points without an obvious choice for numeric feature vectors: scientific papers connected by citations, users of a social network connected by friendships, or web pages connected by links. On the photograph in Fig. 1(a), the nodes are the patches and each patch is connected to the ten patches of most similar color; spectral clustering then separates the five surface types (Fig. 1(e)), since the patches of one surface type are neighbors in color. Fig. 3 arranges the methods by the representation of the data points, as Euclidean vectors or as the nodes of a similarity graph, and of a cluster; the penalty term of convex clustering is weighted by the edges of that graph.

![Figure 3 of the entry clustering](https://dictionaryofml.org/terms/fig/clustering-fig3.svg)

*Figure 3: Clustering methods arranged by the representation of the data points: as Euclidean vectors, or as the nodes of a graph with weighted edges. Methods on the left branch differ in the cluster shapes they can represent: k-means delivers isotropic clusters, round regions around a cluster centroid; a Gaussian mixture model delivers ellipsoids and density-based spatial clustering of applications with noise any shape a chain of nearby data points can trace. Methods on the right branch determine the clusters from the connectivity of the nodes: spectral clustering from eigenvectors of the Laplacian matrix, flow-based clustering from network flows between selected nodes. Convex clustering uses both representations: its data-fidelity term needs the feature vectors of the data points, and its penalty term is weighted by the edge weights of the graph*

Convex clustering uses both representations of Fig. 3. It gives every data point ${\mathbf x}^{(r)}$ its own cluster centroid ${{\mathbf w}}^{(r)}$, which avoids the non-convexity of the clustering error. The method minimizes the convex objective function

$$
\sum_{r=1}^{m}
\left\lVert {\mathbf x}^{(r)} -
{{\mathbf w}}^{(r)} \right\rVert_{2}^{2} + \alpha
\sum_{1 \le r < r' \le m}
A_{r,r'}
\left\lVert {{\mathbf w}}^{(r)} -
{{\mathbf w}}^{(r')} \right\rVert_{2} \text{.}
$$

The penalty term fuses cluster centroids with weights $A_{r,r'} \geq 0$ (Hocking et al., 2011; Sun et al., 2021, Eq. (1); Lindsten et al., 2011). The data points that share a cluster centroid at the unique minimizer form a cluster. At $\alpha = 0$ every cluster centroid equals its data point, so distinct data points form $m$ clusters. Once $\alpha$ is large enough, all cluster centroids of a connected component of the graph with the weights $A_{r,r'}$ as its edge weights coincide, so the number of clusters is set indirectly through $\alpha$. With the edge weights of a similarity graph as weights, the penalty term fuses only neighboring data points and the clusters follow the graph (Hocking et al., 2011). On the photograph, with the edge weights of the color graph of the patches and the $\alpha$ of Fig. 1(f), the fused cluster centroids leave $65$ clusters, four of them with at least $80$ patches; like density-based spatial clustering of applications with noise, the method fuses sky with cloud along the gradual color transition (Fig. 1(f)).

Beyond image segmentation, clustering is used for customer segmentation (Wedel and Kamakura, 2000), for grouping documents by topic in natural language processing (Manning et al., 2008, Sect. 16.1), for community detection in social networks (Fortunato, 2010), and for anomaly detection, where a data point that falls into no cluster or into a tiny one is an outlier (Chandola et al., 2009). As a preprocessing step for supervised learning, clustering reduces a dataset to its cluster centroids as prototypes (Hastie et al., 2009, Sect. 13.2.1), or adds the distances of a data point to the cluster centroids as new features (Coates and Ng, 2012). It can also split the dataset so that a separate hypothesis is learned per cluster, e.g., one linear regression per cluster (Späth, 1979). For a large dataset, k-means is run on a coreset: a small weighted subset of the data points whose weighted average squared Euclidean distance to the nearest cluster centroid approximates that of the full dataset for every choice of the cluster centroids (Har-Peled and Mazumdar, 2004).

**Synonyms:** segmentation.

**See also:** cluster, cluster centroid, clustering error, degree of belonging, hard clustering, soft clustering, [$k$-means](https://dictionaryofml.org/terms/kmeans.html), [Gaussian mixture model](https://dictionaryofml.org/terms/gmm.html), [expectation–maximization](https://dictionaryofml.org/terms/em.html), density-based spatial clustering of applications with noise, spectral clustering, graph clustering, flow-based clustering, convex clustering, coreset, [hyperparameter](https://dictionaryofml.org/terms/hyperparameter.html), anomaly detection, image segmentation, similarity graph, Laplacian matrix, $k$-nearest-neighbor graph, [unsupervised learning](https://dictionaryofml.org/terms/unsupervisedlearning.html).

## References

- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Jain et al. (1999). Data clustering: a review. *ACM Computing Surveys*. <https://doi.org/10.1145/331499.331504>
- Lloyd (1982). Least squares quantization in PCM. *IEEE Trans. Inf. Theory*. <https://doi.org/10.1109/TIT.1982.1056489>
- Ester et al. (1996). A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. *Proceedings of the Second International Conference on Knowledge Discovery and Data Mining*.
- Luxburg (2007). A tutorial on spectral clustering. *Statist. Comput.*. <https://doi.org/10.1007/s11222-007-9033-z>
- Ng et al. (2001). On Spectral Clustering: Analysis and an Algorithm. *Adv. Neural Inf. Process. Syst.*. <https://papers.nips.cc/paper_files/paper/2001/hash/801272ee79cfde7fa5960571fee36b9b-Abstract.html>
- SarcheshmehPour et al. (2021). Flow-Based Clustering and Spectral Clustering: A Comparison. *2021 55th Asilomar Conf. Signals, Syst., Comput.*. <https://doi.org/10.1109/IEEECONF53345.2021.9723162>
- Hocking et al. (2011). Clusterpath: An Algorithm for Clustering using Convex Fusion Penalties. *Proc. of the 28th International Conference on Machine Learning,*.
- Sun et al. (2021). Convex Clustering: Model, Theoretical Guarantee and Efficient Algorithm. *J. Mach. Learn. Res.*. <https://jmlr.org/papers/v22/18-694.html>
- Lindsten et al. (2011). Clustering using sum-of-norms regularization: With application to particle filter output computation. *2011 IEEE Statistical Signal Processing Workshop (SSP)*. <https://doi.org/10.1109/SSP.2011.5967659>
- Wedel and Kamakura (2000). Market Segmentation: Conceptual and Methodological Foundations. *Kluwer Academic Publishers*.
- Manning et al. (2008). Introduction to Information Retrieval. *Cambridge University Press*.
- Fortunato (2010). Community detection in graphs. *Physics Reports*. <https://doi.org/10.1016/j.physrep.2009.11.002>
- Chandola et al. (2009). Anomaly Detection: A Survey. *ACM Comput. Surv.*. <https://doi.org/10.1145/1541880.1541882>
- Coates and Ng (2012). Learning Feature Representations with K-Means. *Neural Networks: Tricks of the Trade*. <https://doi.org/10.1007/978-3-642-35289-8_30>
- Späth (1979). Algorithm 39: Clusterwise linear regression. *Computing*. <https://doi.org/10.1007/BF02265317>
- Har-Peled and Mazumdar (2004). On coresets for $k$-means and $k$-median clustering. *Proc. 36th Annual ACM Symp. Theory of Computing (STOC)*. <https://doi.org/10.1145/1007352.1007400>

## Cite this entry

```bibtex
@misc{dictml_clustering,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {clustering},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/clustering.html}
}
```


---

# confusion matrix

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-04. [HTML](https://dictionaryofml.org/terms/cm.html) · [typeset PDF](https://dictionaryofml.org/terms/cm.pdf) · [LaTeX](https://dictionaryofml.org/terms/cm.tex) · [Python demo](https://dictionaryofml.org/terms/cm.py) · [notebook](https://dictionaryofml.org/terms/cm.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

The confusion matrix of a hypothesis on a finite dataset with $K$ label values is the $K \times K$ matrix whose entry in row $y$ and column $\widehat{y}$ counts the data points with true label $y$ and prediction $\widehat{y}$. Its diagonal counts the correct predictions, and each off-diagonal entry counts one kind of misclassification. The accuracy, the precision, and the recall are obtained from the matrix by normalizing the diagonal, a column, or a row. On imbalanced data, the confusion matrix exposes failure on the rare class that the accuracy alone hides.

## Definition

The weather station in Krems an der Donau (Austria) records the minimum and the maximum air temperature of each day. A linear classifier is learned by logistic regression from a training set $\mathcal{D}^{(\rm train)}$ of $40$ such data points from February and April 2024, and its decision boundary, a straight line, splits the plane of the two temperatures (Fig. 1). The features of a data point are the two temperatures of one day; its label indicates whether the minimum temperature of the following day stays above $0\,^{\circ}\mathrm{C}$ or falls below it, a frost warning.

![Figure 1 of the entry cm](https://dictionaryofml.org/terms/fig/cm-fig1.svg)

*Figure 1: The $40$ data points of the Krems an der Donau frost warning with the linear decision boundary learned by logistic regression. Each data point is a day's minimum and maximum temperature, marked by whether the following day stays above $0\,^{\circ}\mathrm{C}$ (filled circles) or brings frost (open triangles). Data generated by pythondemos/cm.py*

Most days pose no risk: only $6$ of the $40$ following days bring frost. The constant prediction ``above zero'', a baseline that ignores the features, is therefore correct on $34$ of the $40$ days, an accuracy of $34/40 = 0.85$, and the learned classifier reaches the same accuracy of $0.85$. The single number cannot tell the two apart, nor reveal which mistakes either makes. A classifier with that accuracy might miss the frost days or raise false alarms. The confusion matrix breaks the count of predictions down by what was true and what was predicted (Fig. 2). The learned classifier detects one of the six frost days at the cost of one false alarm, while the baseline detects none and raises none.

![Figure 2 of the entry cm](https://dictionaryofml.org/terms/fig/cm-fig2.svg)

*Figure 2: The confusion matrices of the learned classifier (a) and of the always-above-zero baseline (b) of Fig. 1. Both reach an accuracy of $0.85$; only the confusion matrices show that the learned classifier detects one frost day and raises one false alarm, while the baseline detects none and raises none. The shaded diagonal cells hold the correct predictions. Data generated by pythondemos/cm.py*

Consider a finite dataset with $m$ data points, each characterized by a feature vector ${\mathbf x}$ and a label $y$ from a finite label space $\mathcal{Y} = \{1, \,\ldots, \,K\}$. For a given hypothesis $h$, the confusion matrix is a $K \times K$ matrix with one row per true label value and one column per predicted value (Hastie et al., 2009, Sect. 9.1.2). The entry in row $y$ and column $\widehat{y}$ is the number of data points with true label $y$ and prediction $h({\mathbf x}) = \widehat{y}$. A diagonal entry is one with $\widehat{y} = y$, where the prediction agrees with the label, so the diagonal counts the correctly classified data points, and the off-diagonal entries count each kind of misclassification separately.

The standard classification metrics are ratios of entries of the confusion matrix. The accuracy is the sum of the diagonal divided by $m$. Normalizing a row by its sum yields the fraction of data points of that class that are detected (for the positive class, the recall), while normalizing a column yields the fraction of predictions of that class that are right, the precision. For the learned classifier of Fig. 2, with frost as the positive class, the recall is $1/6$ and the precision is $1/2$. The baseline has a recall of $0$, and its precision is not defined at all, since it never predicts frost (pythondemos/cm.py). On imbalanced data, the confusion matrix exposes what a large accuracy can hide. A classifier that never predicts the rare class fills the row of that class with misclassifications while keeping most predictions correct. The always-above-zero baseline in Fig. 2 does exactly that (see baseline).

All numbers above are computed on the training set $\mathcal{D}^{(\rm train)}$, the $40$ days the classifier was learned from. A frost warning is judged on days that did not belong to $\mathcal{D}^{(\rm train)}$, so the confusion matrix, and every ratio taken from it, must be computed again on a validation set before one classifier is preferred over the other.

**See also:** [classification](https://dictionaryofml.org/terms/classification.html), [accuracy](https://dictionaryofml.org/terms/accuracy.html), precision, recall, imbalanced data, [baseline](https://dictionaryofml.org/terms/baseline.html), [training set](https://dictionaryofml.org/terms/trainset.html), [validation set](https://dictionaryofml.org/terms/valset.html), metric, [label](https://dictionaryofml.org/terms/label.html), label space, [matrix](https://dictionaryofml.org/terms/matrix.html).

## References

- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>

## Cite this entry

```bibtex
@misc{dictml_cm,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {confusion matrix},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/cm.html}
}
```


---

# convex

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-07. [HTML](https://dictionaryofml.org/terms/convex.html) · [typeset PDF](https://dictionaryofml.org/terms/convex.pdf) · [LaTeX](https://dictionaryofml.org/terms/convex.tex) · [Python demo](https://dictionaryofml.org/terms/convex.py) · [notebook](https://dictionaryofml.org/terms/convex.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A subset of the Euclidean space is convex if it contains the line segment between any two of its points. A function is convex if its epigraph is a convex set. A convex optimization problem is the minimization of a convex function over a convex set. Empirical risk minimization is a convex optimization problem whenever the hypothesis space is parameterized by a convex set of model parameters and the loss function is convex in those model parameters. Any local minimizer of a convex objective function is necessarily also a global minimizer, so a method that reaches a local minimum has solved the optimization problem.

## Definition

Learning a linear model by gradient descent moves the model parameters step by step, each iteration reducing the objective function within a neighborhood of the current model parameters. Many machine learning methods learn model parameters by such an iterative optimization method, with gradient descent and its variants the most widely used.

These optimization methods are guaranteed to converge to a global minimum when the objective function is a differentiable convex function of the model parameters that attains its minimum and has a Lipschitz continuous gradient (see smooth), and the step size is chosen suitably (Boyd and Vandenberghe, 2004, Sect. 9.3; Bubeck, 2015, Thm. 3.3; Nesterov, 2004, Thm. 2.1.14). For a convex function every local minimum is necessarily a global minimum: any choice of model parameters at which no small change reduces the objective function already attains the smallest value that the objective function takes anywhere. The average squared error loss of linear regression is a convex function of the model parameters, and so is the objective function of the support vector machine. Empirical risk minimization over a hypothesis space parameterized by a convex set of model parameters, with a loss function that is convex in them, is a convex optimization problem: the minimization of a convex function over a convex set.

Convexity is defined both for sets and for functions. For sets, the definition is geometric. A subset $\mathcal{C} \subseteq \mathbb{R}^{d}$ of the Euclidean space $\mathbb{R}^{d}$ is referred to as convex if it contains the line segment between any two of its points (Boyd and Vandenberghe, 2004, Sect. 2.1.4). For ${\mathbf w}, {\mathbf w}' \in \mathcal{C}$, that is

$$
\beta {\mathbf w} + (1-\beta) {\mathbf w}' \in \mathcal{C}
\quad \text{for all } \beta \in [0,1] \text{.}
$$

Similarly, a function $f: \mathbb{R}^{d} \rightarrow \mathbb{R}$ is convex if its epigraph $\big\{ \big( {\mathbf w}^{\top},t \big)^{\top} \in \mathbb{R}^{d+1} : t \geq f({\mathbf w}) \big\}$ is a convex set (Boyd and Vandenberghe, 2004, Sect. 3.1.7), the epigraph being the set of points on or above the graph of $f$. Examples of a convex set and of a convex function are illustrated in Fig. 1.

![Figure 1 of the entry convex](https://dictionaryofml.org/terms/fig/convex-fig1.svg)

*Figure 1: (a) Convex set $\mathcal{C} \subseteq \mathbb{R}^{d}$: the line segment between any two points of $\mathcal{C}$ stays in $\mathcal{C}$. (b) Convex function $f: \mathbb{R}^{d} \rightarrow \mathbb{R}$: the shaded epigraph is a convex set, and the line segment connecting any two points on the graph of $f$ lies above the graph*

Two families of convex sets appear throughout machine learning. The first is the halfspace $\{ {\mathbf w} \in \mathbb{R}^{d} : {\mathbf a}^{\top} {\mathbf w} \leq b \}$. It is determined by a nonzero normal vector ${\mathbf a} \in \mathbb{R}^{d} \setminus \{\mathbf{0}\}$, which fixes the direction of the bounding hyperplane, and an offset $b \in \mathbb{R}$, which fixes its position. If two points satisfy the inequality, so does every point of the segment between them (Boyd and Vandenberghe, 2004, Sect. 2.2.1). The second is the convex hull of finitely many vectors ${\mathbf x}^{(1)}, \ldots, {\mathbf x}^{(m)}$. It is defined as the set of all their convex combinations,

$$
\operatorname{conv}\big\{ {\mathbf x}^{(1)}, \ldots, {\mathbf x}^{(m)} \big\}
:= \Big\{ \sum_{r=1}^{m} \beta_{r} {\mathbf x}^{(r)}
\; : \; \beta_{r} \geq 0 \text{ for } r = 1, \ldots, m,
\; \sum_{r=1}^{m} \beta_{r} = 1 \Big\} \text{.}
$$

It is the smallest convex set that contains these vectors (Boyd and Vandenberghe, 2004, Sect. 2.1.4).

Every closed convex set is given by the intersection of all halfspaces that contain it (Boyd and Vandenberghe, 2004, Sect. 2.3.1; Rockafellar, 1970, Thm. 11.5)

\begin{equation}
\tag{1}
\mathcal{C} = \bigcap \big\{ H \subseteq \mathbb{R}^{d}
\;:\; H \text{ is a halfspace}, \; \mathcal{C} \subseteq H \big\}
\text{.}
\end{equation}

In general, intersecting a subset of the halfspaces containing a convex set yields a convex set that contains the original one, i.e., an outer approximation (see Fig. 2).

![Figure 2 of the entry convex](https://dictionaryofml.org/terms/fig/convex-fig2.svg)

*Figure 2: A convex set $\mathcal{C}$ and two halfspaces $H_{1}, H_{2}$ that contain it, in three shades of gray: lightest where only one halfspace covers the plane, mid-gray for their intersection $H_{1} \cap H_{2}$, darkest for $\mathcal{C}$ itself. Neither halfspace is tight: its bounding hyperplane does not touch $\mathcal{C}$, leaving the gaps marked by the double arrows. The intersection is therefore a strictly larger convex set than $\mathcal{C}$ – an outer approximation*

The halfspace intersection Eq. (1) can be narrowed: it needs to consider only those halfspaces that are bounded by a supporting hyperplane. Every boundary point of a non-empty convex set carries such a hyperplane (Boyd and Vandenberghe, 2004, Sect. 2.5.2), and each point outside the set is already excluded by one of the halfspaces that these supporting hyperplanes bound: for a point ${\mathbf x} \notin \mathcal{C}$ and $\mathcal{C}$ closed, there is a unique nearest point ${\mathbf p} \in \mathcal{C}$ to ${\mathbf x}$ (Boyd and Vandenberghe, 2004, Sect. 8.1). That nearest point is a boundary point of $\mathcal{C}$, and therefore one of the points that carry a supporting hyperplane: a point of the interior could be moved a little toward ${\mathbf x}$ and would still lie in $\mathcal{C}$, so it would not be the nearest one. The hyperplane through ${\mathbf p}$ with normal vector ${\mathbf x} - {\mathbf p}$, i.e., orthogonal to ${\mathbf x} - {\mathbf p}$, supports $\mathcal{C}$ at ${\mathbf p}$, and the halfspace it bounds contains $\mathcal{C}$ but not ${\mathbf x}$ (Boyd and Vandenberghe, 2004, Sects. 2.5.1 and 2.5.2; see Fig. 3). Fig. 4 shows eight supporting hyperplanes of an ellipse.

![Figure 3 of the entry convex](https://dictionaryofml.org/terms/fig/convex-fig3.svg)

*Figure 3: Why the supporting hyperplanes already suffice to recover a closed convex set $\mathcal{C}$, here a disk (darker shading). For a point ${\mathbf x}$ outside $\mathcal{C}$, let ${\mathbf p}$ be the point of $\mathcal{C}$ nearest to ${\mathbf x}$. The hyperplane through ${\mathbf p}$ orthogonal to ${\mathbf x} - {\mathbf p}$ (solid line) touches $\mathcal{C}$ at ${\mathbf p}$, so it is a supporting hyperplane, and the halfspace it bounds (lighter shading) contains $\mathcal{C}$ but not ${\mathbf x}$. Every point outside $\mathcal{C}$ is excluded by one such halfspace, so intersecting these halfspaces recovers $\mathcal{C}$*

![Figure 4 of the entry convex](https://dictionaryofml.org/terms/fig/convex-fig4.svg)

*Figure 4: A convex set $\mathcal{C}$ (shaded ellipse) and eight hyperplanes that touch it (gray lines). Each of them bounds a halfspace that contains $\mathcal{C}$, and the intersection of all halfspaces containing $\mathcal{C}$ is $\mathcal{C}$ itself*

For the epigraph of a differentiable convex function $f$, the supporting hyperplane is determined by the gradient. At the boundary point $\big({\mathbf w}^{\top}, f({\mathbf w})\big)^{\top}$, its normal vector is $\big(\nabla f({\mathbf w})^{\top}, -1\big)^{\top}$, since

$$
\big(\nabla f({\mathbf w})^{\top}, -1\big)
\left[ \begin{pmatrix} {\mathbf w}' \\ t \end{pmatrix}
- \begin{pmatrix} {\mathbf w} \\ f({\mathbf w}) \end{pmatrix} \right]
\leq 0
\quad \text{for every } \big({\mathbf w}'^{\top}, t\big)^{\top} \in \operatorname{epi} f
\text{.}
$$

This is the first-order condition $f({\mathbf w}') \geq f({\mathbf w}) + \nabla f({\mathbf w})^{\top} ({\mathbf w}' - {\mathbf w})$ written as a statement about the epigraph (Boyd and Vandenberghe, 2004, Sect. 3.1.7). At a point where the gradient vanishes, the condition gives $f({\mathbf w}') \geq
f({\mathbf w})$ for every ${\mathbf w}'$, which is the claim above that a local minimum of a convex function is a global one.

The outer approximation of Fig. 2 is also used by approximate inference in probabilistic models defined over a graph. The set of achievable marginal distributions is a convex set described by a large number of linear inequalities. Message-passing methods optimize over an outer approximation of it obtained from a subset of those inequalities (Wainwright and Jordan, 2008, Sect. 3.4.1; Sontag and Jaakkola, 2007).

Of the two families of convex sets introduced above, the convex hull is the one that characterizes the linear separability of data points with binary label values. Consider $m$ data points with feature vectors ${\mathbf x}^{(r)} \in \mathbb{R}^{d}$ and labels $y^{(r)} \in \{-1,+1\}$, for $r = 1, \ldots, m$. These data points are linearly separable if some linear classifier predicts every label correctly, i.e., there are ${\mathbf w} \in \mathbb{R}^{d} \setminus \{\mathbf{0}\}$ and $b \in \mathbb{R}$ with $y^{(r)} \big( {\mathbf w}^{\top} {\mathbf x}^{(r)} + b \big) > 0$ for $r = 1, \ldots, m$. This holds precisely when the convex hull of the feature vectors labeled $+1$ and the convex hull of those labeled $-1$ do not intersect (Boyd and Vandenberghe, 2004, Sects. 2.5.1 and 8.6.1). Separability is therefore a property of two convex sets built from the data points.

Convexity also appears in a widely used family of probabilistic models. An exponential family is constituted by the probability distributions whose probability density function has the form

$$
p({\mathbf x}; {\mathbf w}) = h({\mathbf x})
\exp\big({\mathbf w}^{\top} \mathbf{t}({\mathbf x}) - A({\mathbf w})\big)
\text{.}
$$

This probability density function is parameterized by the vector ${\mathbf w}$ and the sufficient statistics $\mathbf{t}(\cdot)$. The log-partition function $A$ is fully determined by requiring $p({\mathbf x}; {\mathbf w})$ to integrate to one. Its Hessian is the covariance matrix of the sufficient statistics $\mathbf{t}({\mathbf x})$ under $p({\mathbf x}; {\mathbf w})$,

$$
\nabla^{2} A({\mathbf w})
= {\mathbb{E}} \big\{ \mathbf{t}({\mathbf x}) \mathbf{t}({\mathbf x})^{\top} \big\}
- {\mathbb{E}} \big\{ \mathbf{t}({\mathbf x}) \big\}
{\mathbb{E}} \big\{ \mathbf{t}({\mathbf x}) \big\}^{\top}
\text{,}
$$

which is positive semi-definite. A twice-differentiable function whose Hessian is positive semi-definite is convex (Boyd and Vandenberghe, 2004, Sect. 3.1.4), so $A$ is a convex function on its domain (Wainwright and Jordan, 2008, Prop. 3.1). The convexity of $A$ implies that the negative log-likelihood function of a training set, up to an additive constant that does not depend on ${\mathbf w}$,

$$
- \sum_{r=1}^{m} {\mathbf w}^{\top}
\mathbf{t}\big({\mathbf x}^{(r)}\big) + m A({\mathbf w})
\text{,}
$$

is a convex function of ${\mathbf w}$. Thus, for an exponential family, maximum likelihood estimation is a convex optimization problem.

**See also:** epigraph, halfspace, supporting hyperplane, convex optimization, Hessian.

## References

- Boyd and Vandenberghe (2004). Convex Optimization. *Cambridge Univ. Press*. <https://doi.org/10.1017/CBO9780511804441>
- Bubeck (2015). Convex Optimization: Algorithms and Complexity. *Found. Trends Mach. Learn.*. <https://doi.org/10.1561/2200000050>
- Nesterov (2004). Introductory Lectures on Convex Optimization: A Basic Course. *Kluwer Academic*. <https://doi.org/10.1007/978-1-4419-8853-9>
- Rockafellar (1970). Convex Analysis. *Princeton Univ. Press*. <https://doi.org/10.1515/9781400873173>
- Wainwright and Jordan (2008). Graphical Models, Exponential Families, and Variational Inference. *Found. Trends Mach. Learn.*. <https://doi.org/10.1561/2200000001>
- Sontag and Jaakkola (2007). New Outer Bounds on the Marginal Polytope. *Adv. Neural Inf. Process. Syst. (NeurIPS)*. <https://proceedings.neurips.cc/paper_files/paper/2007/hash/2b323d6eb28422cef49b266557dd31ad-Abstract.html>

## Cite this entry

```bibtex
@incollection{dictml_convex,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {convex},
  booktitle = {Dictionary of Applied Machine Learning},
  publisher = {Springer Nature Singapore},
  year = {2026},
  doi = {10.1007/978-981-92-2012-0_18-1},
  isbn = {978-981-92-2012-0},
  url = {https://doi.org/10.1007/978-981-92-2012-0_18-1}
}
```


---

# correlation

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/correlation.html) · [typeset PDF](https://dictionaryofml.org/terms/correlation.pdf) · [LaTeX](https://dictionaryofml.org/terms/correlation.tex) · [Python demo](https://dictionaryofml.org/terms/correlation.py) · [notebook](https://dictionaryofml.org/terms/correlation.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Correlation measures the strength and direction of the linear relationship between two real-valued RVs. It is obtained by normalizing their covariance by their standard deviations, making it dimensionless and comparable across quantities measured on different scales. Its extreme positive and negative values characterize perfect increasing and decreasing affine relationships, while its central value characterizes uncorrelated variables. Independent RVs are uncorrelated, but uncorrelated RVs need not be independent, since correlation quantifies only linear dependence. The correlation between the predictions of the base learners of an ensemble bounds the variance reduction that averaging them can achieve.

## Definition

On a day with an unusually warm morning at Krems an der Donau, the afternoon tends to be unusually warm too. Fig. 1 shows a dataset of $1827$ daily weather measurements at that station. Each data point represents a day, characterized by the minimum and the maximum temperature as its feature vector ${\mathbf x}=(x_{1},x_{2})^{\top}$. Selecting a data point from the dataset uniformly at random defines a random vector ${\mathbf x}$. Correlation quantifies whether a day whose minimum temperature is unusually high also tends to have an unusually high maximum temperature.

More generally, for two real-valued RVs $x$ and $y$ on a common probability space with nonzero standard deviations $\sigma_{x}$ and $\sigma_{y}$, their correlation is their covariance normalized by the product of these standard deviations (Bertsekas and Tsitsiklis, 2008, Sect. 4.2),

$$
\rho := \frac{{\rm cov}\left(x,y\right)}{\sigma_{x}\,\sigma_{y}} \in [-1, 1]\text{.}
$$

The value $\rho$ is also referred to as the Pearson correlation coefficient.

Unlike the covariance, correlation is dimensionless. Adding a constant to either random variable, or multiplying it by a positive constant, does not change $\rho$; multiplying one random variable by a negative constant reverses its sign (Anderson, 2003, Sect. 2.3, Thm. 2.3.2). The extreme values $\rho = 1$ and $\rho = -1$ occur exactly when $y$ is an increasing or a decreasing affine function of $x$ with probability one. Indeed, $|\rho| = 1$ is the case of equality in the Cauchy-Schwarz inequality for the centered RVs $x - \mathbb{E}\{x\}$ and $y - \mathbb{E}\{y\}$, which holds exactly when one is a constant multiple of the other with probability one, and the sign of that constant is the sign of $\rho$. The sample correlation of two data vectors has the same property (Meyer, 2000, Sect. 5.4). The RVs are uncorrelated when $\rho = 0$ (Bertsekas and Tsitsiklis, 2008, Sect. 4.2). Independent RVs are uncorrelated, but uncorrelated RVs need not be independent, since $\rho$ quantifies only linear dependence (Bertsekas and Tsitsiklis, 2008, Sect. 4.2). For the random vector ${\mathbf x}$ obtained by selecting uniformly at random one of the daily minimum-and-maximum temperature pairs shown in Fig. 1, the covariance between the two temperatures is $55.08$ squared degrees Celsius, and their variances are $49.89$ and $83.41$ squared degrees Celsius. Thus,

$$
\rho = \frac{55.08}{\sqrt{49.89\cdot83.41}} \approx 0.85\text{.}
$$

![Figure 1 of the entry correlation](https://dictionaryofml.org/terms/fig/correlation-fig1.svg)

*Figure 1: Daily realizations of the weather random vector ${\mathbf x}$: every sixth day from $2020$ to $2024$. The correlation $\rho\approx0.85$, computed from all $1827$ days, reflects the tendency of high daily minima to accompany high daily maxima. Data generated by pythondemos/correlation.py*

The correlation between the predictions of the base learners of an ensemble limits how much averaging them can help. An average of $M$ identically distributed predictions, each with variance $\sigma^{2}$ and pairwise correlation $\rho$, has variance $\rho \sigma^{2} + (1 - \rho)\sigma^{2} / M$ (Hastie et al., 2009, Sect. 15.2). The second term vanishes as $M$ grows, while the first remains, which is why bagging resamples the training set and a random forest randomizes the features at each split: both lower the correlation between their predictions.

## References

- Bertsekas and Tsitsiklis (2008). Introduction to Probability. *Athena Scientific*.
- Anderson (2003). An Introduction to Multivariate Statistical Analysis. *Wiley*.
- Meyer (2000). Matrix Analysis and Applied Linear Algebra. *SIAM*.
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>

## Cite this entry

```bibtex
@misc{dictml_correlation,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {correlation},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/correlation.html}
}
```


---

# covariance matrix

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/covmtx.html) · [typeset PDF](https://dictionaryofml.org/terms/covmtx.pdf) · [LaTeX](https://dictionaryofml.org/terms/covmtx.tex) · [Python demo](https://dictionaryofml.org/terms/covmtx.py) · [notebook](https://dictionaryofml.org/terms/covmtx.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

The covariance matrix of a random vector consists of the variances of the individual entries on its main diagonal and the covariances between different entries off the diagonal. Together with the mean vector, the covariance matrix collects the second-order statistics of a random vector. For a Gaussian random vector, the mean vector and the covariance matrix determine the linear hypothesis with the smallest risk: the model parameters of this hypothesis and its risk can be read off the entries of the covariance matrix. For a Gaussian random vector, the smallest risk of predicting one entry from the others under the squared error loss is the reciprocal of a diagonal entry of its inverse, the precision matrix.

## Definition

A warm morning at Krems an der Donau usually comes with a warm afternoon, and the mornings vary less than the afternoons. Fig. 1 shows a dataset of $1827$ daily weather measurements at that station. Each data point represents a day, characterized by the minimum and the maximum temperature as its feature vector ${\mathbf x} = (x_{1}, x_{2})^{\top}$. A compact summary of this dataset is the covariance matrix of a random vector obtained by selecting a data point from the dataset uniformly at random. That summary is the sample covariance matrix of the dataset.

Formally, the covariance matrix of a random vector ${\mathbf x} \in \mathbb{R}^{d}$ is defined as the expectation of the outer product of the centered vector ${\mathbf x} - \mathbb{E}\{{\mathbf x}\}$ with itself (if it exists):

$$
\mathbf{C}^{({\mathbf x})} := \mathbb{E} \bigg \{ \big( {\mathbf x} - \mathbb{E} \big\{ {\mathbf x} \big\} \big)
\big({\mathbf x} - \mathbb{E} \big\{ {\mathbf x} \big\} \big)^{\top} \bigg\} \text{.}
$$

The entry in row $j$ and column $j'$ is the covariance of $x_{j}$ and $x_{j'}$. The diagonal entries $\big( \mathbf{C}^{({\mathbf x})} \big)_{j,j}$ are therefore the variances of the individual entries of ${\mathbf x}$. Fig. 1 also shows an ellipse (Boyd and Vandenberghe, 2004, Sect. 2.2.2), obtained as the level set $\big\{ {\mathbf x} : \big( {\mathbf x} - \mathbb{E}\{{\mathbf x}\} \big)^{\top} \big( \mathbf{C}^{({\mathbf x})} \big)^{-1} \big( {\mathbf x} - \mathbb{E}\{{\mathbf x}\} \big) = 4 \big\}$ for the covariance matrix $\mathbf{C}^{({\mathbf x})}$ obtained from the weather measurements. The random vector ${\mathbf x}$ collects the minimum and the maximum temperature of a day selected uniformly at random, so each data point drawn in Fig. 1 is a realization of ${\mathbf x}$. The mean vector $\mathbb{E}\{{\mathbf x}\}$ is the center of the cloud and of the ellipse.

![Figure 1 of the entry covmtx](https://dictionaryofml.org/terms/fig/covmtx-fig1.svg)

*Figure 1: (a) Daily temperature measurements at Krems an der Donau, which can be interpreted as realizations of a random vector ${\mathbf x} \in \mathbb{R}^{2}$: every sixth day of the years $2020$ to $2024$ (gray dots), the mean vector $\mathbb{E}\{{\mathbf x}\}$ of all $1827$ days (cross), and the level set at the value four of the quadratic form in $\big( \mathbf{C}^{({\mathbf x})} \big)^{-1}$ (solid curve). The curve is tilted because $x_{1}$ and $x_{2}$ have a positive covariance. (b) The covariance matrix of all $1827$ days, in squared degrees Celsius: the shaded diagonal entries are the variances of $x_{1}$ and $x_{2}$, and the dashed off-diagonal entries are their covariance. Data generated by pythondemos/covmtx.py*

A covariance matrix is always positive semi-definite, i.e., ${\mathbf v}^{\top} \mathbf{C}^{({\mathbf x})} {\mathbf v} \geq 0$ for every vector ${\mathbf v} \in \mathbb{R}^{d}$, since this quadratic form is the variance of ${\mathbf v}^{\top} {\mathbf x}$. Since it is also symmetric, it admits the spectral decomposition (Horn and Johnson, 2013, Thm. 4.1.5)

$$
\mathbf{C}^{({\mathbf x})} = \mathbf{V} \boldsymbol{\Lambda} \mathbf{V}^{\top},
$$

where the columns of $\mathbf{V}$ are orthonormal eigenvectors and $\boldsymbol{\Lambda}$ is diagonal with the eigenvalues, each of them nonnegative since it is the quadratic form above at its eigenvector. It is invertible, and then positive definite, exactly when all these eigenvalues are positive. Equivalently, no entry of ${\mathbf x}$ is, with probability one, an affine function of the other entries.

Beyond its use as a summary for a dataset, the covariance matrix determines the fundamental limit and the optimal method for predicting one feature from the others. The inverse of an invertible covariance matrix, referred to as the precision matrix (Bishop, 2006, Sect. 2.3.1), answers this task through its diagonal entry $\big( \big( \mathbf{C}^{({\mathbf x})} \big)^{-1} \big)_{j,j}$. The remaining features form the vector ${\mathbf x}_{-j} := \big( x_{1}, \,\ldots, \,x_{j-1}, x_{j+1}, \,\ldots, \,x_{d} \big)^{\top} \in \mathbb{R}^{d-1}$. For a Gaussian random vector ${\mathbf x} \sim \mathcal{N}\left({\boldsymbol \mu},\mathbf{C}^{({\mathbf x})}\right)$ (see multivariate normal distribution), the conditional probability distribution of $x_{j}$, given ${\mathbf x}_{-j}$, is Gaussian with a variance that does not depend on the value of ${\mathbf x}_{-j}$. This conditional variance is $1 \big/ \big( \big( \mathbf{C}^{({\mathbf x})} \big)^{-1} \big)_{j,j}$ (Bishop, 2006, Sect. 2.3.1, Eq. (2.73)).

This conditional variance becomes a bound on every machine learning method once the data points can be approximated as realizations of independent and identically distributed random vectors ${\mathbf x}^{(r)} \sim \mathcal{N}\left({\boldsymbol \mu},\mathbf{C}^{({\mathbf x})}\right)$. The conditional variance is also the smallest risk that any hypothesis can achieve under the squared error loss,

\begin{equation}
\tag{1}
\min_{h: \mathbb{R}^{d-1} \to \mathbb{R}}
\mathbb{E} \Big\{ \big( x_{j}
- h\big({\mathbf x}_{-j}\big) \big)^{2} \Big\}
= 1 \big/ \big( \big( \mathbf{C}^{({\mathbf x})} \big)^{-1}
\big)_{j,j} \text{.}
\end{equation}

This smallest risk is the Bayes risk of this regression problem. It is attained by the Bayes estimator $\mathbb{E}\{ x_{j} \mid {\mathbf x}_{-j} \}$, which depends linearly on ${\mathbf x}_{-j}$ in the Gaussian case (Bertsekas and Tsitsiklis, 2008, Sect. 8.3; Bishop, 2006, Sect. 2.3.1). For the weather measurements of Fig. 1, the best linear hypothesis predicting the minimum temperature of a day from its maximum temperature has the risk $13.5$ squared degrees Celsius, against the variance $49.9$ of the minimum temperature, which is the risk of the best constant prediction. Its two model parameters are computed from $\mathbf{C}^{({\mathbf x})}$ and $\mathbb{E}\{{\mathbf x}\}$: the slope is the ratio of the covariance to the variance of the maximum temperature, and the intercept is fixed by the two means (Bertsekas and Tsitsiklis, 2008, Sect. 8.4) (see linear regression). Under a Gaussian probabilistic model this risk is also the Bayes risk. By the Bayes risk identity Eq. (1), the diagonal of $\big( \mathbf{C}^{({\mathbf x})} \big)^{-1}$ yields a baseline.

The spectral decomposition also yields a feature transformation that removes linear correlations. If $\mathbf{C}^{({\mathbf x})}$ is invertible, define its inverse square root as

$$
\big(\mathbf{C}^{({\mathbf x})}\big)^{-1/2}
:= \mathbf{V} \boldsymbol{\Lambda}^{-1/2} \mathbf{V}^{\top},
$$

where $\boldsymbol{\Lambda}^{-1/2}$ is obtained by replacing every eigenvalue with its inverse square root. The transformed feature vector

$$
\widetilde{{\mathbf x}}
:= \big(\mathbf{C}^{({\mathbf x})}\big)^{-1/2}
\big({\mathbf x}-\mathbb{E}\{{\mathbf x}\}\big)
$$

has zero mean and covariance matrix $\mathbf{C}^{(\widetilde{{\mathbf x}})}=\mathbf{I}$. Thus, its entries have unit variance and are mutually uncorrelated, as illustrated in Fig. 2. This transformation is known as whitening (Bishop, 2006, Sect. 12.1.3, Eqs. (12.24)–(12.25)). By removing differences in scale and linear correlations between the features, whitening can improve the conditioning of optimization problems and thereby speed up gradient descent and related gradient-based methods. For linear regression with the squared error loss on centered features, the Hessian of the risk is $2 \mathbf{C}^{({\mathbf x})}$, and one gradient descent step with an exact line search shrinks the gap to the smallest risk of a linear hypothesis by a factor of at most $1 - 1/\kappa\!\left(\mathbf{C}^{({\mathbf x})}\right)$, where the condition number $\kappa\!\left(\mathbf{C}^{({\mathbf x})}\right)$ is the ratio of the largest to the smallest eigenvalue (Boyd and Vandenberghe, 2004, Sect. 9.3.1). After whitening, $\mathbf{C}^{(\widetilde{{\mathbf x}})} = \mathbf{I}$, this ratio is one, and a single step reaches the minimum. In practice, the mean and covariance matrix used for this transformation are estimated solely from the training set.

![Figure 2 of the entry covmtx](https://dictionaryofml.org/terms/fig/covmtx-fig2.svg)

*Figure 2: The weather data of Fig. 1 before (left) and after (right) multiplication of each centered feature vector by $\big(\mathbf{C}^{({\mathbf x})}\big)^{-1/2}$. The covariance ellipse becomes a circle, and the transformed features have unit variance and zero covariance.*

**See also:** covariance, [matrix](https://dictionaryofml.org/terms/matrix.html), [random variable](https://dictionaryofml.org/terms/rv.html), random vector, [variance](https://dictionaryofml.org/terms/variance.html), sample covariance matrix, multivariate normal distribution, spectral decomposition, [eigenvalue decomposition](https://dictionaryofml.org/terms/evd.html), [eigenvalue](https://dictionaryofml.org/terms/eigenvalue.html), [linear regression](https://dictionaryofml.org/terms/linreg.html), Bayes risk, [baseline](https://dictionaryofml.org/terms/baseline.html).

## References

- Boyd and Vandenberghe (2004). Convex Optimization. *Cambridge Univ. Press*. <https://doi.org/10.1017/CBO9780511804441>
- Horn and Johnson (2013). Matrix Analysis. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781139020411>
- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Bertsekas and Tsitsiklis (2008). Introduction to Probability. *Athena Scientific*.

## Cite this entry

```bibtex
@misc{dictml_covmtx,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {covariance matrix},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/covmtx.html}
}
```


---

# data

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-10-29. [HTML](https://dictionaryofml.org/terms/data.html) · [typeset PDF](https://dictionaryofml.org/terms/data.pdf) · [LaTeX](https://dictionaryofml.org/terms/data.tex) · [Python demo](https://dictionaryofml.org/terms/data.py) · [notebook](https://dictionaryofml.org/terms/data.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Data are representations of information, recorded in a form suitable for storage, communication and processing, and the word carries three technical senses in machine learning. In its general sense, it denotes recorded information in a form suitable for processing by machine learning methods. As a data point, it is an elementary information-carrying unit; as a dataset, it is an indexed collection of data points. A related regulatory term, personal data, is defined in the general data protection regulation and adopted by the EU AI Act.

## Definition

A weather station measures the temperature at its location and stores the value $23.4\,^{\circ}\mathrm{C}$ together with a timestamp (see Fig. 1(a)). Such recorded values are data: representations of information, recorded in a form suitable for storage, communication, and processing by ML systems (International Organization for Standardization and International Electrotechnical Commission, 2015). Every machine learning method trains a model on data, and the usefulness of the hypothesis the method learns is limited by the quality and quantity of the available data.

The word *data* is used in three technical senses, summarized in Fig. 1, and additionally in a legal sense. In its broadest use, data refer to recorded information. This sense applies when it does not matter whether a single data point or a collection of them is meant. Examples are raw sensor measurements such as the temperature reading in Fig. 1(a) and compound terms such as data parallelism, networked data, and data augmentation.

As a formal unit, a data point is the fundamental information-carrying unit (Fig. 1(b)). Its information is contained in two types of properties: features, which are easy to measure or compute, and labels, which are difficult to measure or compute or need human annotation. The formal model of a domain set, a label set and a labeling function that assigns each domain point its label is given by Shalev-Shwartz and Ben-David (2014, Sect. 2.1).

As a formal collection, a dataset $\mathcal{D} = \{{\mathbf z}^{(1)}, \ldots, {\mathbf z}^{(m)}\}$ is the indexed collection of data points on which model training and validation are performed (Fig. 1(c)) (Goodfellow et al., 2016, Sect. 5.1). The quality and the quantity of data that limit a learned hypothesis are properties of this collection: the labels of its data points and their number.

![Figure 1 of the entry data](https://dictionaryofml.org/terms/fig/data-fig1.svg)

*Figure 1: Three technical meanings of the word *data* in machine learning applications. (a) Data in the general sense: a temperature reading sampled at time instant $t^{*}$. (b) A data point ${\mathbf z}=(x,y)$ with a single feature $x$ and label $y$. (c) A dataset that consists of a finite number of data points*

The word *data* also carries a regulatory meaning. The general data protection regulation defines personal data as any information relating to an identified or identifiable natural person (European Parliament and Council of the European Union, 2016, Art. 4(1)). The EU AI Act adopts that definition and defines biometric data as personal data resulting from specific technical processing (European Parliament and Council of the European Union, 2024, Art. 3(34) and 3(50)). Several regulatory obligations on ML systems, including the data minimization principle, the right to erasure, and the right to explanation, refer to data in this legal sense.

**See also:** [data point](https://dictionaryofml.org/terms/datapoint.html), [dataset](https://dictionaryofml.org/terms/dataset.html), personal data, general data protection regulation, EU AI Act.

## References

- International Organization for Standardization and International Electrotechnical Commission (2015). Information technology --- Vocabulary. <https://www.iso.org/obp/ui/#iso:std:iso-iec:2382:ed-1:v2:en>
- Goodfellow et al. (2016). Deep Learning. *MIT Press*.
- European Parliament and Council of the European Union (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (Text with EEA relevance). <https://eur-lex.europa.eu/eli/reg/2016/679/oj>
- European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act) (Text with EEA relevance). <https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng>
- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>

## Cite this entry

```bibtex
@incollection{dictml_data,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {data},
  booktitle = {Dictionary of Applied Machine Learning},
  publisher = {Springer Nature Singapore},
  year = {2026},
  doi = {10.1007/978-981-92-2012-0_1-1},
  isbn = {978-981-92-2012-0},
  url = {https://doi.org/10.1007/978-981-92-2012-0_1-1}
}
```


---

# data augmentation

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-07. [HTML](https://dictionaryofml.org/terms/dataaug.html) · [typeset PDF](https://dictionaryofml.org/terms/dataaug.pdf) · [LaTeX](https://dictionaryofml.org/terms/dataaug.tex) · [Python demo](https://dictionaryofml.org/terms/dataaug.py) · [notebook](https://dictionaryofml.org/terms/dataaug.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Data augmentation enlarges a training set with synthetic data points obtained from transformations or perturbations of existing ones. Typical examples are rotating or cropping an image and adding small perturbations to the features and labels of a measured data point. If the label does not change under a transformation, training on transformed copies makes the learned hypothesis vary little under it. Sensing devices measure with limited precision, so perturbed copies reflect the uncertainty interval around each recorded value. Data augmentation is a form of regularization, beside model pruning and adding a penalty term to the training error. For linear regression, perturbing the features with zero-mean noise whose covariance matrix is a multiple of the identity is, in expectation over the noise, equivalent to adding the ridge regression penalty term.

## Definition

Three measured days at Krems an der Donau are all a forecaster has to learn from. Each day is a data point, characterized by the maximum daytime temperature as its feature $x$, and the maximum daytime temperature of the next day as its label $y$ (Fig. 1). The figure also shows a learned hypothesis map $\widehat{h}$, obtained by empirical risk minimization using the polynomials of degree two as hypothesis space.

The learned hypothesis achieves zero training error but delivers poor predictions outside the training set, which is overfitting. On a held-out set of $125$ data points obtained from other days of 2024, whose feature lies inside the measured range, its average squared error loss is $256.2$ squared degrees. One reason for the failure of $\widehat{h}$ is that the training set is too small. For almost every training set of sample size $m = 3$ there is one polynomial of degree two passing through its data points.

Data augmentation enlarges a dataset with synthetic data points, obtained by perturbations of the features and labels of the measured ones. Sensing devices have imperfections such as thermal noise, so a reading $x$ is one representative of an uncertainty interval $\mathcal{U}^{(x)}$. Any other value from that interval could have been observed instead.

Considering the weather data from Fig. 1, a temperature reading $x$ carries $\mathcal{U}^{(x)} = [x - 1, x + 1]$. Each measured day $\left( x,y \right)$ is replaced by $400$ copies $\left( x + \varepsilon,y + \varepsilon' \right)$, whose perturbations $\varepsilon$ and $\varepsilon'$ are realizations of RVs distributed uniformly on $[-1, 1]$. The augmented training set $\widetilde{\mathcal{D}}$ holds $1200$ data points, none of which required a new temperature measurement. The width of the uncertainty interval used for the perturbation of the features and labels is a hyperparameter. One basic way to determine it is to try a few choices and compare the validation error of the resulting learned hypotheses.

Fig. 1 also shows the polynomial of degree two $\widehat{h}'$ that empirical risk minimization learns using $\widetilde{\mathcal{D}}$ as training set. The hypothesis $\widehat{h}'$ no longer passes through the three measured days, but it delivers better predictions outside the training set. Its training error on them is $2.8$ squared degrees, but its average squared error loss on the held-out data points falls to $24.7$. The coefficient of $x^{2}$ drops from $\widehat{w}_{2} = 0.71$ for $\widehat{h}$ to $\widehat{w}_{2}' = 0.06$ for $\widehat{h}'$.

![Figure 1 of the entry dataaug](https://dictionaryofml.org/terms/fig/dataaug-fig1.svg)

*Figure 1: Three data points (filled circles) that represent daily weather conditions at Krems an der Donau. The solid curve is the degree-two polynomial $\widehat{h}$ with a training error of zero on them. It dips far below every temperature that was measured. The open squares are copies drawn from the uncertainty interval of each measured day, five of the copies of every day. The dashed curve is the hypothesis $\widehat{h}'$ learned from them. Data generated by pythondemos/dataaug.py*

Enlarging $\mathcal{D}$ with perturbed copies is one basic form of regularization, beside model pruning, which shrinks the hypothesis space, and adding a penalty term to the training error. These different forms of regularization are sometimes equivalent in their effect on the learned hypothesis (Fig. 2). For linear regression, adding zero-mean perturbations with covariance matrix $\sigma^{2} \mathbf{I}$ to each feature vector yields the ridge regression solution, asymptotically in the number of copies. Its penalty term is $\sigma^{2} \left\lVert {\mathbf w} \right\rVert_{2}^{2}$ for the weight vector ${\mathbf w}$ (Bishop, 1995; Goodfellow et al., 2016, Sect. 7.5).

![Figure 2 of the entry dataaug](https://dictionaryofml.org/terms/fig/dataaug-fig2.svg)

*Figure 2: The three forms of regularization on the data points of Fig. 1. Only the feature is perturbed here, the case the equivalence with ridge regression covers, so each open square sits beside its measured day at the same label. The double arrow marks the width of the uncertainty interval they are drawn from. Enlarging the training set with them replaces the solid curve $\widehat{h}$ by the dashed one, which stays within $0.10\,^{\circ}$C of the $\widehat{h}'$ of Fig. 1. Adding the penalty term $\alpha \mathcal{R}\big\{ h \big\}$ to the training error, shown below the plot, is the second form. Shrinking the hypothesis space to $\{h : |w_{2}| \leq 0.06\}$ is the third; the hypothesis it yields lands within $0.20\,^{\circ}$C of the dashed curve and is not drawn separately. Data generated by pythondemos/dataaug.py*

Instead of adding noise to the features and labels, perturbations can also come from a transformation $\mathcal{T}$ of the feature space. A transformation known to leave the label alone is a symmetry of the learning task. A rotated or cropped image of a cat still shows a cat, while its feature vector, obtained by stacking the pixel color intensities, changes. The transformation must preserve the label, and a rotation does so only up to a point. Fig. 3 reduces images of the handwritten digits 6 and 9 to their first two principal directions (see principal component analysis) as features $x_{1}, x_{2}$. A small rotation $\mathcal{T}$ keeps an image of a 6 among the other 6s. However, rotating an image by $180$ degrees carries it into the 9s, where the label has changed.

![Figure 3 of the entry dataaug](https://dictionaryofml.org/terms/fig/dataaug-fig3.svg)

*Figure 3: Images of the handwritten digits 6 (circles) and 9 (triangles), each represented by two features $x_{1}, x_{2}$ obtained from principal component analysis. The solid line with square marks is the path that one image of a 6 traces as it is rotated from $0$ to $180$ degrees. The three inlays above the plot are that image upright, turned by $30$ degrees, and turned by $180$ degrees, each placed above its own point of the path. Its first steps keep the image among the 6s, so a rotation by a small angle preserves the label. Its last steps place the image among the 9s. Data generated by pythondemos/dataaug.py*

**See also:** [regularization](https://dictionaryofml.org/terms/regularization.html), model pruning, penalty term, ridge regression, [overfitting](https://dictionaryofml.org/terms/overfitting.html), [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), [training set](https://dictionaryofml.org/terms/trainset.html), [dataset](https://dictionaryofml.org/terms/dataset.html), [hypothesis space](https://dictionaryofml.org/terms/hypospace.html), [hypothesis](https://dictionaryofml.org/terms/hypothesis.html), [data point](https://dictionaryofml.org/terms/datapoint.html), [feature](https://dictionaryofml.org/terms/feature.html), [label](https://dictionaryofml.org/terms/label.html), [feature vector](https://dictionaryofml.org/terms/featurevec.html), feature space, uncertainty, [hyperparameter](https://dictionaryofml.org/terms/hyperparameter.html), [validation error](https://dictionaryofml.org/terms/valerr.html), polynomial regression, learning task.

## References

- Bishop (1995). Training with Noise is Equivalent to Tikhonov Regularization. *Neural Computation*. <https://doi.org/10.1162/neco.1995.7.1.108>
- Goodfellow et al. (2016). Deep Learning. *MIT Press*.

## Cite this entry

```bibtex
@misc{dictml_dataaug,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {data augmentation},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/dataaug.html}
}
```


---

# data imputation

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/dataimputation.html) · [typeset PDF](https://dictionaryofml.org/terms/dataimputation.pdf) · [LaTeX](https://dictionaryofml.org/terms/dataimputation.tex) · [Python demo](https://dictionaryofml.org/terms/dataimputation.py) · [notebook](https://dictionaryofml.org/terms/dataimputation.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Data imputation replaces missing attribute values of data points by estimates computed from the attribute values that were recorded. In tabular data a missing value is an empty cell of the table, and imputation fills the cell. It is one response to missing data, alongside discarding the incomplete data points. The filling is a machine learning problem: the missing value is the label of a data point, the recorded values beside it are its features, and the complete data points form a training set on which a hypothesis map (an imputer) is learned.

## Definition

The weather station Krems an der Donau records, for each day, the minimum temperature, the precipitation, and the maximum temperature. The year $2024$ is therefore a table with one row per day and one column per quantity (GeoSphere Austria, 2026). This table is tabular data. A day on which the thermometer failed leaves the maximum-temperature cell of its row empty, while the table keeps its shape (Fig. 1). Data imputation fills such a cell: it replaces missing attribute values of data points, features or labels, by estimates computed from the attribute values that were recorded (Little and Rubin, 2019). It is one response to missing data, alongside discarding the incomplete data points or using a method that tolerates the gaps (Little and Rubin, 2019).

![Figure 1 of the entry dataimputation](https://dictionaryofml.org/terms/fig/dataimputation-fig1.svg)

*Figure 1: Four of the $366$ rows of the table of daily weather measurements recorded in $2024$, July 9–12. Each row is a data point, namely one day; the headers abbreviate the minimum temperature (min. temp.), the precipitation (precip.), and the maximum temperature (max. temp.), and the special value $-1.0$ of the precipitation column records that no precipitation was observed. The maximum-temperature cells of July 10 and 11 are empty. The light gray column holds the feature and the dark gray column the label of a day, so the two complete rows shown, and the other complete days, supply both, while the two empty cells are the labels to be imputed*

The filling is a machine learning problem (Abayomi et al., 2008). The empty cell is the label of its row, and the recorded cells of the same row are its features. The complete rows form a training set, since each one supplies both a feature vector and the label that belongs with it. A hypothesis map, an imputer, is learned from them by empirical risk minimization. Applying that hypothesis to an incomplete row yields the estimate that fills its cell. The simplest imputation fills every empty cell of a column with the sample mean of the recorded cells of that column, which is the hypothesis that ignores the feature vector (Little and Rubin, 2019).

A numerical experiment takes $15$ days spread over the year and empties the maximum-temperature cells of $3$ of them. From the $12$ complete rows, it learns a linear regression hypothesis that maps the minimum temperature of a day to its maximum temperature. The emptied cells are then filled with its predictions, shown as a scatterplot in Fig. 2. The removed values are known, so the filling can be evaluated. The average squared error loss on the $3$ cells, in squared degrees Celsius, is $17.6$ for the learned hypothesis and $150.2$ for the column average.

![Figure 2 of the entry dataimputation](https://dictionaryofml.org/terms/fig/dataimputation-fig2.svg)

*Figure 2: Scatterplot of $15$ days of the table in Fig. 1, spread over the year, with the minimum temperature of a day as its feature and the maximum temperature as its label. The $12$ complete days are gray circles, and the solid line is the linear regression hypothesis learned from them. For each of the $3$ emptied cells, a cross marks the removed value and an open square the imputed value*

A missing value can sit among other missing values, so that the feature vector of its row is itself incomplete. Rectangular patches lost from an aerial photograph are such a case. Each pixel is a data point, its features are the colors of the eight pixels around it, and its label is its own color, so a lost pixel inside a patch has lost neighbors as well. The filling then proceeds in sweeps: the missing values start at the sample mean of the recorded ones, and each sweep re-estimates them from the values that their neighbors currently hold, until the values no longer change. For a linear model, one sweep maps the vector ${\mathbf x}$ of missing values to ${\mathbf M} {\mathbf x} + {\mathbf b}$, with a matrix ${\mathbf M}$ built from the weights of the hypothesis. Two runs started at ${\mathbf x}$ and ${\mathbf x}'$ differ after $t$ sweeps by ${\mathbf M}^{t} ({\mathbf x} - {\mathbf x}')$, so the sweeps converge from every start if and only if ${\mathbf M}^{t} \to \mathbf{0}$, i.e., if and only if the spectral radius $\rho({\mathbf M})$ is below one (Horn and Johnson, 2013, Thm. 5.6.12), and the limit is then the unique fixed point of the sweep. A numerical experiment fills the lost pixels by such sweeps. For the hypothesis learned from the intact pixels by least squares, $\rho({\mathbf M}) =
2.72$ and the sweeps diverge. The penalty term of ridge regression shrinks the weights: with the regularization parameter $\alpha = 0.03$, the hypothesis learned by ridge regression gives $\rho({\mathbf M}) = 0.84$, and the sweeps converge. With that hypothesis, the average squared error loss on the lost pixels is half that of the baseline that fills every one of them with the average color, and the boundaries where the colors change sharply come out blurred.

An imputed value is an estimate and not a measurement, and a method trained on imputed data points cannot tell the two apart. Part of the training set was produced by a hypothesis rather than observed, so a training error that averages over imputed cells scores a hypothesis partly against the predictions of the imputer rather than against measurements; for the imputer itself, every filled cell contributes zero loss. A single filled value also carries no information about the uncertainty of the estimate (Hastie et al., 2009, Sect. 9.6; Little and Rubin, 2019). Recording which values were imputed, and keeping them out of the test set, keeps the estimate of the risk free of values that a hypothesis produced.

**Synonyms:** imputation, missing-value imputation.

**See also:** missing data, [tabular data](https://dictionaryofml.org/terms/tabulardata.html), [data point](https://dictionaryofml.org/terms/datapoint.html), [feature](https://dictionaryofml.org/terms/feature.html), [label](https://dictionaryofml.org/terms/label.html), [feature vector](https://dictionaryofml.org/terms/featurevec.html), [training set](https://dictionaryofml.org/terms/trainset.html), [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), [linear regression](https://dictionaryofml.org/terms/linreg.html), squared error loss, [baseline](https://dictionaryofml.org/terms/baseline.html), [test set](https://dictionaryofml.org/terms/testset.html).

## References

- GeoSphere Austria (2026). GeoSphere Austria climate station archive. <https://dataset.api.hub.geosphere.at/v1/station/historical/klima-v2-1d>
- Little and Rubin (2019). Statistical Analysis with Missing Data. *Wiley*. <https://doi.org/10.1002/9781119482260>
- Abayomi et al. (2008). Diagnostics for Multivariate Imputations. *J. Roy. Statist. Soc.: Ser. C (Appl. Statist.)*. <https://doi.org/10.1111/j.1467-9876.2007.00613.x>
- Horn and Johnson (2013). Matrix Analysis. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781139020411>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>

## Cite this entry

```bibtex
@misc{dictml_dataimputation,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {data imputation},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/dataimputation.html}
}
```


---

# data leakage

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-10-29. [HTML](https://dictionaryofml.org/terms/dataleakage.html) · [typeset PDF](https://dictionaryofml.org/terms/dataleakage.pdf) · [LaTeX](https://dictionaryofml.org/terms/dataleakage.tex) · [Python demo](https://dictionaryofml.org/terms/dataleakage.py) · [notebook](https://dictionaryofml.org/terms/dataleakage.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Data leakage arises when a machine learning method uses features that are available during training but not at prediction time. For example, in weather forecasting each data point covers a time interval, and one feature is the temperature one hour ahead of the forecast time. On historic data this later measurement is available, but when a forecast is actually computed it is not, so the feature leaks information. A leaked feature makes the loss on the training set and test set look near zero, yet the learned hypothesis fails after deployment, because the feature cannot be reproduced at prediction time. The term is also used, in a security sense, for the unintentional release of sensitive information.

## Definition

Consider a machine learning method trained on an hourly temperature sequence, where each data point is a sliding window of three consecutive hours (Fig. 1). The label is the temperature of the middle hour, and one of the features is the measurement one hour later. On historic data this later measurement is available, so the loss on the training set and test set looks near zero; but in real time the middle hour must be predicted before the next hour is measured, so the learned hypothesis is useless. This is data leakage: a feature used during training or evaluation is unavailable when a prediction must be delivered.

![Figure 1 of the entry dataleakage](https://dictionaryofml.org/terms/fig/dataleakage-fig1.svg)

*Figure 1: Hourly temperature measurements. Each data point is a sliding window of three hours (box); the label is the middle hour's temperature. Using the measurement one hour later as a feature leaks information: at prediction time (dashed line) that hour has not been measured yet, so the feature is available for historic training data only*

More generally, a leaked feature is one whose value cannot be reproduced at prediction time, whether it is measured only later (as above) or computed from the label. Data leakage is formalized by Kaufman et al. (2012), who also give methods to detect and avoid it. Kapoor and Narayanan (2023) survey it as a cause of the reproducibility crisis in machine learning-based science. Test set contamination, the presence of test set data points in the training set, is a related form of data leakage: there, the information that should be unavailable at training time is the test set itself.

The term *data leakage* is also used, in an unrelated security sense, for the unintentional release of sensitive information (data exfiltration from a trained system), a meaning distinct from the methodological one defined above.

**See also:** test set contamination, [training set](https://dictionaryofml.org/terms/trainset.html), [validation set](https://dictionaryofml.org/terms/valset.html), [test set](https://dictionaryofml.org/terms/testset.html), [generalization](https://dictionaryofml.org/terms/generalization.html), risk.

## References

- Kaufman et al. (2012). Leakage in Data Mining: Formulation, Detection, and Avoidance. *ACM Transactions on Knowledge Discovery from Data*. <https://doi.org/10.1145/2382577.2382579>
- Kapoor and Narayanan (2023). Leakage and the Reproducibility Crisis in Machine-Learning-Based Science. *Patterns*. <https://doi.org/10.1016/j.patter.2023.100804>

## Cite this entry

```bibtex
@misc{dictml_dataleakage,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {data leakage},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/dataleakage.html}
}
```


---

# data point

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/datapoint.html) · [typeset PDF](https://dictionaryofml.org/terms/datapoint.pdf) · [LaTeX](https://dictionaryofml.org/terms/datapoint.tex) · [Python demo](https://dictionaryofml.org/terms/datapoint.py) · [notebook](https://dictionaryofml.org/terms/datapoint.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A data point is the elementary information-carrying unit on which machine learning methods operate. The properties of a data point fall into two categories: features, which are easily measurable or computable, and labels, which are higher-level facts that typically can only be determined using human expertise. Whether a given property is treated as a feature or a label is a design choice that depends on the machine learning application.

## Definition

An image of a cow herd (Fig. 1), a patient in a clinic, and a protein are objects that a machine learning method processes. A data point is an object that conveys information (Cover and Thomas, 2006); such objects are the elementary units on which machine learning methods operate. Examples include students, radio signals, trees, images, RVs, real numbers, or proteins. Data points of the same type are described by two categories of properties.

The first category includes features that are measurable or computable properties of a data point. They can be automatically extracted or computed using sensors, computers, or other data collection systems. The second category includes labels that are higher-level facts, or quantities of interest, that typically require human expertise or domain knowledge to determine rather than being directly measurable. Machine learning aims to predict the label of a data point from its features.

Fig. 1 shows an image as an example of a data point. Several properties of the image can serve as features: the color intensities $x_{1}, \ldots, x_{d}$ of all image pixels, the timestamp $x_{d+1}$ of the image capture, and the spatial location $x_{d+2}$ of the image capture. Higher-level facts can serve as labels: the number of cows $y_{1}$, the number of wolves $y_{2}$, and the condition of the pasture $y_{3}$ (e.g., healthy or overgrazed).

![Figure 1 of the entry datapoint](https://dictionaryofml.org/terms/fig/datapoint-fig1.svg)

*Figure 1: Illustration of a data point consisting of an image. Different properties of the image can serve as features, such as the average color intensities of the image pixels. Higher-level facts about the image, such as the presence of different object categories, can serve as labels*

For a data point that represents a patient, a feature could be the body weight, while the label could be the presence of cancer, the fact of interest. Determining this label requires a medical expert (or even a committee of experts) to examine the patient. A property treated as a label in one setting (e.g., a cancer diagnosis) may serve as a feature in another, where reliable automation (e.g., image analysis) allows the property to be computed without human intervention.

Both kinds of properties are error-prone. For many machine learning applications, it is rarely possible to access the true labels. For example, a medical expert's diagnosis can be wrong, making it a noisy proxy for the patient's true condition. Errors of this kind, called label noise, arise whenever a label is only a proxy for the true quantity of interest, whether that proxy is produced by human judgment or by an automated system.

Features are also a source of error within a machine learning method. A feature is obtained by measuring or computing a property of a data point. Physical and chemical sensing devices have a finite measurement uncertainty (Kimothi, 2002), and computing a feature on a finite computer introduces rounding errors (Golub and Loan, 2013, Sect. 2.7). These imperfections make the recorded value deviate from the true feature value. This discrepancy is feature noise.

Machine learning methods should be robust to both feature and label noise. Their predictions on new data points should not degrade sharply when some features or labels in the training set are corrupted. The EU AI Act reflects this concern: training data must, to the best extent possible, be ``free of errors'' (European Parliament and Council of the European Union, 2024, Art. 10(3)), and high-risk AI systems must attain an appropriate level of robustness (European Parliament and Council of the European Union, 2024, Art. 15(1)).

Robustness to feature and label noise can be improved by data augmentation, which deliberately perturbs the features or the label of a training data point (see Fig. 2). Training on such perturbed data points acts as regularization: for the squared error loss and small feature noise, it is equivalent to adding a penalty on the derivatives of the hypothesis with respect to the features (Bishop, 1995, Sect. 3, Eq. (18)).

![Figure 2 of the entry datapoint](https://dictionaryofml.org/terms/fig/datapoint-fig2.svg)

*Figure 2: Data augmentation surrounds a data point (filled circle) with a cloud of perturbed copies (open circles). It is a form of regularization that makes a machine learning method less sensitive to feature and label noise*

**See also:** [data](https://dictionaryofml.org/terms/data.html), [feature](https://dictionaryofml.org/terms/feature.html), [label](https://dictionaryofml.org/terms/label.html), [dataset](https://dictionaryofml.org/terms/dataset.html), [data augmentation](https://dictionaryofml.org/terms/dataaug.html), robustness, EU AI Act.

## References

- Cover and Thomas (2006). Elements of Information Theory. *Wiley*. <https://doi.org/10.1002/047174882X>
- Kimothi (2002). The Uncertainty of Measurements: Physical and Chemical Metrology Impact and Analysis. *American Society for Quality (ASQ)*.
- Golub and Loan (2013). Matrix Computations. *The Johns Hopkins Univ. Press*.
- European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act) (Text with EEA relevance). <https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng>
- Bishop (1995). Training with Noise is Equivalent to Tikhonov Regularization. *Neural Computation*. <https://doi.org/10.1162/neco.1995.7.1.108>

## Cite this entry

```bibtex
@incollection{dictml_datapoint,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {data point},
  booktitle = {Dictionary of Applied Machine Learning},
  publisher = {Springer Nature Singapore},
  year = {2026},
  doi = {10.1007/978-981-92-2012-0_9-1},
  isbn = {978-981-92-2012-0},
  url = {https://doi.org/10.1007/978-981-92-2012-0_9-1}
}
```


---

# dataset

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-10-29. [HTML](https://dictionaryofml.org/terms/dataset.html) · [typeset PDF](https://dictionaryofml.org/terms/dataset.pdf) · [LaTeX](https://dictionaryofml.org/terms/dataset.tex) · [Python demo](https://dictionaryofml.org/terms/dataset.py) · [notebook](https://dictionaryofml.org/terms/dataset.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A dataset is a finite collection of data points on which model training and validation are performed. Strictly speaking, a dataset is a set of distinct data points. With slight abuse of terminology, the term is also used interchangeably with sample, a finite sequence that may contain repetitions. Read as such a sequence, a dataset represents a probability distribution through its empirical distribution, in which a repeated data point counts as often as it is listed. The size of a dataset is measured most simply by its number of data points, which overstates what correlated data points carry and says nothing about how well they cover the feature space. In practice, machine learning methods do not access the data points themselves but a representation of them, such as a table of the relational model, which records their attributes and nothing about the dataset's fitness for a purpose.

## Definition

The cows of Fig. 2, or the rows of Table 1 that record them, form a dataset. A dataset is a collection $\mathcal{D} = \{{\mathbf z}^{(1)}, \ldots, {\mathbf z}^{(m)}\}$ of data points. Machine learning methods use a dataset for model training and validation. Strictly speaking, a dataset is an unordered collection of distinct data points, i.e., a set with no repetitions. In machine learning literature, however, the term is often used for a sample: a sequence of $m$ data points, indexed by their position $r = 1, \ldots, m$ in the sequence, that may contain repetitions (see Fig. 1).

![Figure 1 of the entry dataset](https://dictionaryofml.org/terms/fig/dataset-fig1.svg)

*Figure 1: Two readings of the word *dataset*. (a) Strictly, a dataset is a set of distinct data points, with no order and no repetitions. (b) In machine learning literature, dataset often denotes a sample: a finite sequence of data points indexed by $r$ that may repeat (positions $r = 1$ and $r = 3$ carry the same data point)*

Read as a list, a dataset also represents a probability distribution. Its empirical distribution $\mathbb{P}^{(\mathcal{D})}$ assigns the probability $1/m$ to each entry of the list, so a data point that is listed twice receives the probability $2/m$. A set, which cannot list a data point twice, has no such repetitions. The repetitions are therefore essential wherever a dataset represents a probability distribution. The bootstrap, e.g., draws new datasets from $\mathbb{P}^{(\mathcal{D})}$ and so lists some data points several times.

The size of a dataset has more than one measure. The most basic one is the number $m$ of data points in the list. When the data points are realizations of correlated RVs, however, $m$ overstates what they carry: the average over $30$ consecutive days of a weather record scatters as much as the average over $7$ days drawn at random from the whole record, because consecutive days are correlated (see sample size). The size of a dataset is then defined only relative to a probabilistic model of the data generation, as the number of independent and identically distributed data points that would carry the same information. A third measure asks how well the data points cover the feature space: the largest distance from a point of the region of interest to the nearest data point (see covering), which is small for a dataset whose features are spread out and large for one that leaves parts of the feature space empty, however many data points it holds.

The notion of a dataset is broad: data points may represent concrete physical entities (such as humans or animals) or abstract objects (such as numbers). For illustration, Fig. 2 shows a dataset whose data points are cows.

![Figure 2 of the entry dataset](https://dictionaryofml.org/terms/fig/dataset-fig2.svg)

*Figure 2: Cow herd somewhere in the Alps*

In machine learning applications, the dataset itself is often not accessible. For instance, accessing the dataset in Fig. 2 would require visiting the cow herd. What a machine learning method processes instead is a representation of the dataset within some data model, a formalism for representing and processing data (Abiteboul et al., 1995; Hoberman, 2009; Ramakrishnan and Gehrke, 2002; Silberschatz et al., 2019). For example, the relational model organizes data as a collection of tables, or relations (Codd, 1970; Silberschatz et al., 2019).

A single table consists of rows and columns, where each row corresponds to a single data point and each column represents a specific attribute of a data point. The order of rows does not matter, and each attribute is associated with a domain that specifies its set of allowed values. Machine learning methods use these attributes as the features or the label of a data point, so the attribute domains correspond to the feature space and the label space. Table 1 shows an example obtained from the dataset in Fig. 2.

![Table 1 of the entry dataset](https://dictionaryofml.org/terms/fig/dataset-tab1.svg)

*Table 1: \centering \scshape TABLE \thetable [0.5ex] \scshape Relation (or Table) Representing a Dataset of Cows*

A table records the content of a dataset, but not whether that content is appropriate for a given purpose. A dataset used in ethical and lawful ML systems must meet further requirements. For example, the EU AI Act requires that datasets on which high-risk ML systems are trained be ``sufficiently representative'' of the intended use cases (European Parliament and Council of the European Union, 2024, Art. 10(3)).

Trustworthy artificial intelligence also requires documenting how a dataset was assembled, for example through a datasheet that records the dataset's motivation, composition, and collection process (Gebru et al., 2021). When the data points are personal data, this becomes a legal duty: under the purpose limitation principle, the general data protection regulation requires that such data be collected for ``specified, explicit and legitimate purposes'' (European Parliament and Council of the European Union, 2016, Art. 5(1)(b)). These purposes are fixed at the time of collection.

**See also:** [data point](https://dictionaryofml.org/terms/datapoint.html), sample, [data](https://dictionaryofml.org/terms/data.html), relational model, [feature](https://dictionaryofml.org/terms/feature.html), empirical distribution, [bootstrap](https://dictionaryofml.org/terms/bootstrap.html), [sample size](https://dictionaryofml.org/terms/samplesize.html), covering, EU AI Act.

## References

- Abiteboul et al. (1995). Foundations of Databases. *Addison-Wesley*.
- Hoberman (2009). Data Modeling Made Simple: A Practical Guide for Business and IT Professionals. *Technics Publications*.
- Ramakrishnan and Gehrke (2002). Database Management Systems. *McGraw-Hill*.
- Silberschatz et al. (2019). Database System Concepts. *McGraw-Hill Education*.
- Codd (1970). A Relational Model of Data for Large Shared Data Banks. *Commun. ACM*. <https://doi.org/10.1145/362384.362685>
- European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act) (Text with EEA relevance). <https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng>
- Gebru et al. (2021). Datasheets for datasets. *Commun. ACM*. <https://doi.org/10.1145/3458723>
- European Parliament and Council of the European Union (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (Text with EEA relevance). <https://eur-lex.europa.eu/eli/reg/2016/679/oj>

## Cite this entry

```bibtex
@incollection{dictml_dataset,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {dataset},
  booktitle = {Dictionary of Applied Machine Learning},
  publisher = {Springer Nature Singapore},
  year = {2026},
  doi = {10.1007/978-981-92-2012-0_10-1},
  isbn = {978-981-92-2012-0},
  url = {https://doi.org/10.1007/978-981-92-2012-0_10-1}
}
```


---

# decision boundary

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/decisionboundary.html) · [typeset PDF](https://dictionaryofml.org/terms/decisionboundary.pdf) · [LaTeX](https://dictionaryofml.org/terms/decisionboundary.tex) · [Python demo](https://dictionaryofml.org/terms/decisionboundary.py) · [notebook](https://dictionaryofml.org/terms/decisionboundary.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

The decision boundary of a classifier consists of the feature vectors that lie arbitrarily close to two or more decision regions. An arbitrarily small perturbation of such a feature vector can therefore change the predicted label. Many classifiers threshold a real-valued scoring function, and the decision boundary is then a level set of that function. Its shape depends on the model: a linear classifier has a hyperplane as its decision boundary, a decision tree one made of axis-parallel line segments, and logistic regression on polynomial features a curved one.

## Definition

A frost warning predicts from the minimum and the maximum air temperature of a day whether the following night brings frost. The pairs of temperatures at which the warning switches between frost and no frost form its decision boundary. In general, a classifier is a hypothesis $h\colon\mathcal{X}\rightarrow\mathcal{Y}$ with a finite label space $\mathcal{Y}$. Its decision boundary consists of the feature vectors that are arbitrarily close to two or more of its decision regions. More precisely, ${\mathbf x}\in\mathcal{X}$ lies on the decision boundary if every neighborhood of ${\mathbf x}$ contains feature vectors ${\mathbf x}'$ and ${\mathbf x}''$ such that $h({\mathbf x}')\neq h({\mathbf x}'')$. An arbitrarily small perturbation of such a feature vector can therefore change the predicted label.

In the notation of decision regions, each decision region $\mathcal{R}_{\widehat{y}} := \{{\mathbf x} \in \mathcal{X} :
h({\mathbf x}) = \widehat{y}\}$ is a subset of $\mathcal{X} = \mathbb{R}^{d}$. Its boundary $\partial \mathcal{R}_{\widehat{y}}$ is the set of feature vectors such that every neighborhood contains elements of $\mathcal{R}_{\widehat{y}}$ and of its complement $\mathcal{X} \setminus \mathcal{R}_{\widehat{y}}$. The decision boundary is the union $\bigcup_{\widehat{y} \in \mathcal{Y}} \partial \mathcal{R}_{\widehat{y}}$ of these boundaries over all label values.

Many classifiers are built from a real-valued scoring function $g\colon\mathcal{X}\rightarrow\mathbb{R}$: the predicted label depends on whether $g({\mathbf x})$ lies below or above a threshold $c$. If $g$ is continuous, the decision boundary is the level set $\{{\mathbf x}:g({\mathbf x})=c\}$, provided that $g$ takes values on both sides of $c$ arbitrarily close to every point in this set. For a linear classifier with $g({\mathbf x})={\mathbf w}^{\top}{\mathbf x}$ and ${\mathbf w}\neq\mathbf{0}$, this level set is the hyperplane ${\mathbf w}^{\top}{\mathbf x}=c$, whose normal vector is ${\mathbf w}$ (Bishop, 2006, Sect. 4.1; Hastie et al., 2009, Ch. 4).

Fig. 1 shows that the shape of a decision boundary depends on the model. A linear classifier has a straight decision boundary, a decision tree has one made of axis-parallel line segments (Hastie et al., 2009, Sect. 9.2.1), and logistic regression on polynomial features can have a curved one (Hastie et al., 2009, Sect. 4.1).

![Figure 1 of the entry decisionboundary](https://dictionaryofml.org/terms/fig/decisionboundary-fig1.svg)

*Figure 1: Decision regions learned from the same ten-point training set by three classifiers. Gray marks the region assigned the label $+1$; the thick black lines are the learned decision boundaries. Open circles have label $-1$, and filled squares have label $+1$. The experiment is generated by pythondemos/decisionboundary.py*

**See also:** [hypothesis](https://dictionaryofml.org/terms/hypothesis.html), [feature vector](https://dictionaryofml.org/terms/featurevec.html), label space, boundary, [decision region](https://dictionaryofml.org/terms/decisionregion.html), neighborhood.

## References

- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>

## Cite this entry

```bibtex
@misc{dictml_decisionboundary,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {decision boundary},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/decisionboundary.html}
}
```


---

# decision region

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/decisionregion.html) · [typeset PDF](https://dictionaryofml.org/terms/decisionregion.pdf) · [LaTeX](https://dictionaryofml.org/terms/decisionregion.tex) · [Python demo](https://dictionaryofml.org/terms/decisionregion.py) · [notebook](https://dictionaryofml.org/terms/decisionregion.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A decision region of a classifier is the set of all feature vectors that the classifier maps to one particular label value. The decision regions partition the feature space, one region per label value, and all feature vectors in the same region obtain the same prediction, so the decision regions determine the classifier completely. The union of the boundaries of the decision regions is the decision boundary. The shape of the decision regions depends on the model: a linear classifier has halfspaces, a decision tree has unions of axis-parallel rectangles, and logistic regression on polynomial features has regions separated by a curve.

## Definition

A frost warning predicts from the minimum and the maximum air temperature of a day whether the following night brings frost. Any rule that gives this warning splits the pairs of temperatures into two parts: the pairs for which it predicts frost and the pairs for which it does not. Each part is a decision region of the rule. In general, a classifier is a hypothesis $h\colon
\mathcal{X} \rightarrow \mathcal{Y}$ with a finite label space $\mathcal{Y}$. Its decision region for the label value $\widehat{y} \in \mathcal{Y}$ is the set of feature vectors that it maps to this value,

$$
\mathcal{R}_{\widehat{y}} := \big\{ {\mathbf x} \in
\mathcal{X} : h({\mathbf x}) = \widehat{y}
\big\} \text{.}
$$

The decision regions partition the feature space: every feature vector lies in exactly one of them, there is one region per label value, and a region may consist of several separate pieces (Bishop, 2006, Sect. 1.5.1). All feature vectors in the same decision region obtain the same prediction, so the decision regions determine the classifier completely. The boundary $\partial \mathcal{R}_{\widehat{y}}$ of a decision region is the set of feature vectors such that every neighborhood contains elements of $\mathcal{R}_{\widehat{y}}$ and of its complement. The union of these boundaries over all label values is the decision boundary.

The shape of the decision regions depends on the model. A linear classifier with $h({\mathbf x}) =
\operatorname{sign}({\mathbf w}^{\top} {\mathbf x} + w_{0})$ has the two halfspaces on either side of the hyperplane ${\mathbf w}^{\top} {\mathbf x} + w_{0} = 0$ as its decision regions (Bishop, 2006, Sect. 4.1.1). For more than two label values, a linear classifier predicts the value with the largest of $K$ linear scores. Each decision region is then an intersection of $K - 1$ halfspaces and therefore convex (Bishop, 2006, Sect. 4.1.2). A decision tree compares one feature with a threshold at each split, so its decision regions are unions of axis-parallel rectangles (Hastie et al., 2009, Sect. 9.2.1). Logistic regression on polynomial features has decision regions separated by a curve, the level set of a polynomial.

Fig. 1 shows the decision regions of three classifiers learned from the same data points with two features: a linear classifier, a decision tree, and logistic regression on polynomial features. A data point whose label differs from the label value of the region containing its feature vector is misclassified, and the fraction of such data points is the average 0/1 loss. The two label values are separated by a curve and not by any straight line, so the linear classifier misclassifies half of the data points, while the other two misclassify none.

![Figure 1 of the entry decisionregion](https://dictionaryofml.org/terms/fig/decisionregion-fig1.svg)

*Figure 1: Decision regions of three classifiers learned from the same ten data points with two features. Filled squares are data points with label $y = +1$, and open circles are data points with label $y = -1$. The shaded area is the decision region $\mathcal{R}_{+1}$, the unshaded area the decision region $\mathcal{R}_{-1}$, and the border between them the decision boundary. (a) A linear classifier, whose regions are two half-planes. (b) A decision tree of depth four, whose regions are unions of axis-parallel rectangles. (c) Logistic regression on all monomials of the two features up to degree two, whose regions are separated by a curve. Data generated by pythondemos/decisionregion.py*

**See also:** classifier, [classification](https://dictionaryofml.org/terms/classification.html), [binary classification](https://dictionaryofml.org/terms/binclass.html), [decision boundary](https://dictionaryofml.org/terms/decisionboundary.html), boundary, neighborhood, [linear classifier](https://dictionaryofml.org/terms/linclass.html), [decision tree](https://dictionaryofml.org/terms/decisiontree.html), [logistic regression](https://dictionaryofml.org/terms/logreg.html), halfspace, hyperplane, feature space, label space, $\bf 0/1$ loss.

## References

- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>

## Cite this entry

```bibtex
@misc{dictml_decisionregion,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {decision region},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/decisionregion.html}
}
```


---

# decision tree

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/decisiontree.html) · [typeset PDF](https://dictionaryofml.org/terms/decisiontree.pdf) · [LaTeX](https://dictionaryofml.org/terms/decisiontree.tex) · [Python demo](https://dictionaryofml.org/terms/decisiontree.py) · [notebook](https://dictionaryofml.org/terms/decisiontree.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A decision tree is a flowchart-like representation of a hypothesis map: a root node receives the feature vector of a data point and evaluates a basic test on it. The test result selects a child node, which applies a further test, and the forwarding continues until the feature vector reaches a leaf node. A leaf node has no children and represents one specific function value, i.e., a prediction. The tree partitions the feature space into decision regions, one per leaf node, on which the represented hypothesis is constant. A maximum depth and a family of node tests define the model, the hypothesis space of all such trees, which has no vector of model parameters of fixed length and is therefore searched greedily rather than by gradient methods. Decision trees serve as base models of the random forest and of gradient-boosted decision tree methods. Small decision trees yield interpretable machine learning if the user can comprehend the tests executed at their nodes.

## Definition

A bank's loan approval can be written as a chain of simple tests on the applicant's features that ends in an ``approve'' or ``reject'' prediction. For example, is the income above a threshold? Is the credit score large enough? A decision tree is such a chain in general form: a flowchart-like representation of a hypothesis map $h$.

More formally, a decision tree is a directed acyclic graph whose root node receives the feature vector ${\mathbf x}$ of a data point. The root node then forwards the data point to one of its child nodes based on some elementary test on the features ${\mathbf x}$. If the receiving child node is not a leaf node, i.e., it has child nodes itself, it represents another test. Based on the test result, the data point is forwarded to one of its descendants. This testing and forwarding of the data point is continued until the data point ends up in a leaf node without any children, which carries the prediction $\widehat{y} = h({\mathbf x})$ returned by the tree for this data point (Fig. 1). The leaf nodes partition the feature space into decision regions; in Fig. 1, each test compares an entry of ${\mathbf x}$ with a threshold, so the decision regions are rectangles (Hastie et al., 2009, Sect. 9.2.1). Next to each node, Fig. 1 highlights the part of the feature space that contains the feature vectors reaching that node.

![Figure 1 of the entry decisiontree](https://dictionaryofml.org/terms/fig/decisiontree-fig1.svg)

*Figure 1: Decision tree as a flowchart-like representation of a piecewise constant hypothesis $h: \mathcal{X} \rightarrow \mathbb{R}$ for feature vectors ${\mathbf x} = (x_{1}, x_{2})^{\top}$. Each test compares a single feature with one of the thresholds $\varepsilon, \varepsilon' > 0$, which are marked on the axes of each square. Next to each node, the shaded part of the feature space (depicted as a square with the axes $x_{1}$ and $x_{2}$) contains the feature vectors that reach that node. The shaded parts at the leaf nodes are the decision regions $\mathcal{R}_{\widehat{y}} := \big\{ {\mathbf x} \in \mathcal{X}: h({\mathbf x}) = \widehat{y} \big\}$: rectangles, since every test involves a single feature*

The depth of a decision tree is the number of tests on the longest path from the root node to a leaf node; the tree in Fig. 1 has depth two. Fixing a maximum depth $D$ and a family of allowed tests defines the model: the hypothesis space $\mathcal{H}$ of all decision trees of depth at most $D$ whose node tests belong to the family, for instance all threshold tests $x_{j} \leq
\varepsilon$ with $j \in \{1, \ldots, d\}$ and $\varepsilon \in \mathbb{R}$ (Shalev-Shwartz and Ben-David, 2014, Sect. 18.1). A tree in $\mathcal{H}$ is specified by its shape, by the index $j$ of the feature and the threshold $\varepsilon$ that each test node compares, and by the value of each leaf node.

This hypothesis space has no natural parametrization. The shape of the tree and the index $j$ of the feature tested at each node are discrete choices, and how many of them a tree has depends on its shape, so $\mathcal{H}$ is not the image of a vector of model parameters of fixed length: there is no parameter space $\mathbb{R}^{n}$ on which the training error is a function. The thresholds form a vector once the shape and the tested features are fixed, but the training error is piecewise constant in each of them, since a data point changes child only when the threshold crosses its feature value, so the gradient is zero wherever it exists. Empirical risk minimization over $\mathcal{H}$ can therefore not use gradient descent, and a fixed point view of the training iteration does not apply: an iteration adds a node, it does not update a vector. Minimizing a bound that adds a size penalty to the training error is computationally hard, and the methods below search $\mathcal{H}$ greedily (Shalev-Shwartz and Ben-David, 2014, Sect. 18.2).

Training a decision tree amounts to extending a leaf node by attaching another decision tree. In the simplest case, a leaf node is replaced with a test node whose children are new leaf nodes (Fig. 2).

![Figure 2 of the entry decisiontree](https://dictionaryofml.org/terms/fig/decisiontree-fig2.svg)

*Figure 2: Training grows a decision tree. The tree on the right is obtained from the tree on the left by replacing the leaf node with prediction $\widehat{y}_2$ with a test node whose children are two new leaf nodes (dashed edges)*

Widely used training methods for decision trees include classification and regression trees (CART) (Breiman et al., 1984), iterative dichotomiser 3 (ID3) (Quinlan, 1986), and C4.5 (Quinlan, 1993). Starting from a single leaf node, CART repeatedly replaces a leaf node with a test node. An impurity measure assigns a nonnegative number to a set of labels: it is zero when the labels agree, and it grows as they spread out. Among candidate tests that compare a single feature with a threshold, CART scores each test by the impurity of the two children, weighted by the fraction of data points routed to each of them, and greedily selects the test with the smallest score, i.e., with the largest reduction against the impurity of the parent node. Common impurity measures are the Gini index and the entropy for classification, and the squared error around the mean label for regression (Hastie et al., 2009, Sects. 9.2.2 and 9.2.3). The growth stops only when a leaf node contains too few data points, not when no candidate test reduces the impurity much, since a test that reduces the impurity little can enable a test with a large impurity reduction below it (Hastie et al., 2009, Sect. 9.2.2). The large tree is then pruned back by cost-complexity pruning, which removes the subtrees whose impurity reduction is smaller than the penalty for the leaf nodes they add (Hastie et al., 2009, Sect. 9.2.2).

Fig. 3 shows, as a scatterplot, the result of this training on real data. Each day of $2024$ at the Finnish Meteorological Institute weather station Helsinki Kaisaniemi (Finnish Meteorological Institute, 2024) is a data point, with the minimum temperature of the day as its feature and the maximum temperature as its label. The learned depth-$2$ tree is a piecewise constant hypothesis with four pieces, one per leaf node, each equal to the mean label of the days routed there. The tree was grown to depth two rather than until the stopping rule above applied, so that each of its four pieces is visible in the picture.

![Figure 3 of the entry decisiontree](https://dictionaryofml.org/terms/fig/decisiontree-fig3.svg)

*Figure 3: Each of the $366$ days of $2024$ at the Finnish Meteorological Institute station Helsinki Kaisaniemi is a data point: its feature is the minimum temperature of the day (horizontal axis) and its label the maximum temperature of the day (vertical axis). The open circles are $20$ of them, every $19$th day. The thick solid step function is the piecewise constant hypothesis $\widehat{h}$ learned from all $366$ days by a regression tree of depth two. The three thresholds of its test nodes, at $-10.35\,^{\circ}$C, $6.20\,^{\circ}$C, and $10.55\,^{\circ}$C (dotted verticals), are the positions of the three jumps and split the feature axis into four intervals. On each interval, the step function is flat at the mean label of the data points routed to that leaf node. Data generated by pythondemos/decisiontree.py*

Decision trees serve as base models of the random forest and of gradient-boosted decision tree methods (Hastie et al., 2009, Chs. 10 and 15). A decision tree with a small number of nodes is often considered interpretable: its computation can be traced by following the data point from the root node to a leaf node (Molnar, 2025). This makes small decision trees a common choice in interpretable machine learning.

**See also:** [decision region](https://dictionaryofml.org/terms/decisionregion.html), decision rule, [classification](https://dictionaryofml.org/terms/classification.html), [random forest](https://dictionaryofml.org/terms/randomforest.html), gradient-boosted decision tree, [feature importance](https://dictionaryofml.org/terms/featureimportance.html), [interpretable machine learning](https://dictionaryofml.org/terms/interpretableml.html).

## References

- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Breiman et al. (1984). Classification and Regression Trees. *Wadsworth*.
- Quinlan (1986). Induction of decision trees. *Machine Learning*. <https://doi.org/10.1023/A:1022643204877>
- Quinlan (1993). C4.5: Programs for Machine Learning. *Morgan Kaufmann*.
- Finnish Meteorological Institute (2024). The Finnish Meteorological Institute's open data. <https://opendata.fmi.fi/wfs>
- Molnar (2025). Interpretable Machine Learning: A Guide for Making Black Box Models Explainable. *Ebook*. <https://christophm.github.io/interpretable-ml-book/>

## Cite this entry

```bibtex
@misc{dictml_decisiontree,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {decision tree},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/decisiontree.html}
}
```


---

# dimensionality reduction

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/dimred.html) · [typeset PDF](https://dictionaryofml.org/terms/dimred.pdf) · [LaTeX](https://dictionaryofml.org/terms/dimred.tex) · [Python demo](https://dictionaryofml.org/terms/dimred.py) · [notebook](https://dictionaryofml.org/terms/dimred.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Dimensionality reduction uses a map $h: \mathbb{R}^{d} \to \mathbb{R}^{d'}$ to replace a raw feature vector ${\mathbf x} = (x_{1}, \,\ldots,
\,x_{d})^{\top}$ by a shorter feature vector ${\mathbf z} = (z_{1}, \,\ldots, \,z_{d'})^{\top}$, with $d' < d$. Using fewer features lowers the effective dimension of a model, which reduces the chance of overfitting. Moreover, it lowers the amount of computation needed for model training and for inference. It also enables visualization: two features can be used as the coordinates in a scatterplot. Principal component analysis learns a linear map by minimizing the reconstruction error, while a random projection uses a linear map drawn at random, without any training. An autoencoder learns a nonlinear map represented by an artificial neural network.

## Definition

A square patch of the photograph in Fig. 2 is described by $48$ numbers, and two numbers computed from them suffice to tell sky from vegetation. Dimensionality reduction uses a map $h$, a transformation $h: \mathbb{R}^{d} \rightarrow \mathbb{R}^{d'}$ to replace the raw feature vector ${\mathbf x} = (x_{1}, \,\ldots, \,x_{d})^{\top}$ of a data point by a shorter feature vector ${\mathbf z} = (z_{1}, \,\ldots, \,z_{d'})^{\top}$, with $d' < d$ (Fig. 1). Using a shorter feature vector is beneficial in several ways:

- *Statistical benefit:* It typically reduces the chance of overfitting, as reducing the number of features often reduces the effective dimension of a model (Hastie et al., 2009, Sect. 7.6).
- *Computational benefit:* Using fewer features lowers the amount of computation needed for model training and for inference. For example, linear regression methods need to invert a matrix whose size is determined by the number of features (Hastie et al., 2009, Eq. (3.6)).
- *Visualization:* A map can be learned that delivers two features $z_{1},z_{2}$, which can then be used as the coordinates in a scatterplot. Each square patch of a photograph is first averaged down to a $4 \times 4$ grid of colors, still too many numbers to plot; Fig. 2 then places each patch at the two features that principal component analysis learns for it, which keep most of the variance of the raw feature vectors (Bishop, 2006, Sect. 12.1).

![Figure 1 of the entry dimred](https://dictionaryofml.org/terms/fig/dimred-fig1.svg)

*Figure 1: Dimensionality reduction learns a map $h$ that sends the raw feature vector ${\mathbf x}$ of a data point, a list of $d$ numbers, to a shorter feature vector ${\mathbf z}$ of $d' < d$ numbers*

![Figure 2 of the entry dimred](https://dictionaryofml.org/terms/fig/dimred-fig2.svg)

*Figure 2: Top: the photograph of the Oetscher massif with the patch grid drawn on it; each $128 \times 128$ pixel patch is one data point. Bottom: every patch placed at the two features $z_{1}, z_{2}$ that principal component analysis learns for it. The raw feature vector of a patch is a $4 \times 4$ grid of colors, $48$ numbers; the two learned features keep $86$ percent of its variance. A patch is drawn as an open circle when its blue channel exceeds the other two by a fixed margin (sky) and as a filled triangle when its green channel does (vegetation); the remaining patches, the rock face and the mixed ones along the tree line, are small dots. Sky and vegetation lie in separate regions, with $93$ percent of them on their own side of the midpoint between the two centers. Data generated by pythondemos/dimred.py*

Dimensionality reduction methods differ in the hypothesis space used for the map, in the criterion they learn it by, and in whether the map is learned from data at all. Principal component analysis learns a linear map: the map is a matrix chosen so that the reconstruction error, the squared distance between ${\mathbf x}$ and the feature vector reconstructed from ${\mathbf z}$ by a second linear map, is minimal on average over the training set (Bishop, 2006, Sect. 12.1.2; Hotelling, 1933). An autoencoder learns a nonlinear map represented by an artificial neural network, under the same criterion (Goodfellow et al., 2016, Sect. 14.1). A random projection uses a random matrix without any training: by the Johnson–Lindenstrauss lemma, the distances between the feature vectors of the data points are approximately preserved (Shalev-Shwartz and Ben-David, 2014, Lemma 23.4; Vershynin, 2018). Nonlinear methods beyond the autoencoder include the manifold-learning family, among them t-SNE and UMAP, which are used mainly for visualization (Ghojogh et al., 2023; Meilă and Zhang, 2024).

**See also:** [overfitting](https://dictionaryofml.org/terms/overfitting.html), [autoencoder](https://dictionaryofml.org/terms/autoencoder.html), random projection, [principal component analysis](https://dictionaryofml.org/terms/pca.html), Johnson–Lindenstrauss lemma, [feature vector](https://dictionaryofml.org/terms/featurevec.html), [hypothesis space](https://dictionaryofml.org/terms/hypospace.html).

## References

- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Hotelling (1933). Analysis of a Complex of Statistical Variables into Principal Components. *Journal of Educational Psychology*.
- Goodfellow et al. (2016). Deep Learning. *MIT Press*.
- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Vershynin (2018). High-Dimensional Probability: An Introduction with Applications in Data Science. *Cambridge Univ. Press*.
- Ghojogh et al. (2023). Elements of Dimensionality Reduction and Manifold Learning. *Springer*. <https://doi.org/10.1007/978-3-031-10602-6>
- Meilă and Zhang (2024). Manifold learning: what, how, and why. *Annu. Rev. Stat. Appl.*. <https://doi.org/10.1146/annurev-statistics-040522-115238>

## Cite this entry

```bibtex
@misc{dictml_dimred,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {dimensionality reduction},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/dimred.html}
}
```


---

# distribution shift

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-11. [HTML](https://dictionaryofml.org/terms/distshift.html) · [typeset PDF](https://dictionaryofml.org/terms/distshift.pdf) · [LaTeX](https://dictionaryofml.org/terms/distshift.tex) · [Python demo](https://dictionaryofml.org/terms/distshift.py) · [notebook](https://dictionaryofml.org/terms/distshift.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A data point carries features and a label, drawn together from one joint probability distribution. Distribution shift is a mismatch between that joint probability distribution during training and the one the learned hypothesis is applied to after deployment. The average loss on a test set estimates the risk under the training probability distribution, so a small test error is no evidence about the loss incurred after deployment. A joint probability distribution has two components, a marginal distribution and a conditional probability distribution, and the named special cases differ in which of them changes: covariate shift changes the marginal distribution of the features alone, label shift the marginal distribution of the labels alone, and concept drift the conditional probability distribution. Sample weighting corrects covariate shift when both probability distributions are known; online learning applies when they are not.

## Definition

The Finnish Meteorological Institute (FMI) station Helsinki Kaisaniemi records the minimum and the maximum air temperature of each day. Every summer day is a data point: its feature is the minimum temperature of that day, its label the maximum. The generation of these data points can be approximately modeled as RVs with a joint probability distribution $\mathbb{P}$ of feature $x$ and label $y$. A linear hypothesis $\widehat{h}$, learned by linear regression from $62$ of the $92$ data points of June–August 2026, i.e., by minimizing the average squared error loss, reaches a training error of $5.0$ and a test error of $6.5$ on a test set that consists of the remaining $30$ data points.

The learned hypothesis $\widehat{h}$ is then applied to data points $\mathcal{D}'$ collected at the northernmost FMI station, Utsjoki Nuorgam, during December 2025–February 2026. As with the training set, the generation of these data points can also be approximately modeled as RVs with some joint probability distribution $\mathbb{P}'$. The resulting average loss on $\mathcal{D}'$ is $242.1$, which is $37$ times larger than its test error (Fig. 1). The reason for this discrepancy between test error and the performance on $\mathcal{D}'$ is a distribution shift.

![Figure 1 of the entry distshift](https://dictionaryofml.org/terms/fig/distshift-fig1.svg)

*Figure 1: Distribution shift between the data points generated at two different FMI stations. Each marker is one data point, its feature the daily minimum and its label the daily maximum temperature. The markers indicate every sixth data point of the training set (filled circles), every fourth of the test set (open squares), and every fifth after deployment (open diamonds). The line depicts the hypothesis $\widehat{h}$ learned via linear regression from the training set. The features of the data points encountered after deployment lie outside the range covered by the training set. Moreover, the learned hypothesis $\widehat{h}$ overestimates every label after deployment, by $14.6\,^{\circ}$C on average. Data generated by pythondemos/distshift.py*

Distribution shift refers to a discrepancy between the probability distribution $\mathbb{P}$ underlying the generation of the data points used for training and the probability distribution $\mathbb{P}'$ underlying the generation of the data points to which the learned hypothesis is applied after deployment (Quiñonero-Candela et al., 2009). A distribution shift cannot be detected from the test error when the test set is generated by $\mathbb{P}$ as well. The average loss of $\widehat{h}$ on such a test set is an unbiased estimator of the risk $\mathbb{E}_{\mathbb{P}}\!\big\{ L\left({\mathbf z},\widehat{h} \right) \big\}$ under $\mathbb{P}$, which may differ substantially from the risk $\mathbb{E}_{\mathbb{P}'}\!\big\{
L\left({\mathbf z},\widehat{h} \right) \big\}$ under $\mathbb{P}'$.

The risk under $\mathbb{P}'$ is the one that matters after deployment, and no test set drawn from $\mathbb{P}$ measures it. A small test error is therefore no evidence about the loss incurred under $\mathbb{P}' \neq \mathbb{P}$. Consider a hypothesis that predicts by comparing the single feature with a threshold (this is a simple decision tree). The two pdfs in Fig. 2, those of a standard normal distribution $\mathbb{P}$ and of a normal distribution $\mathbb{P}'$ with a larger mean and standard deviation, show the effect of replacing $\mathbb{P}$ by $\mathbb{P}'$ on a threshold chosen under $\mathbb{P}$. The fraction of data points beyond the threshold rises from $14\%$ to $88\%$, so the risk under $\mathbb{P}'$ exceeds the risk under $\mathbb{P}$ that the test error estimates.

![Figure 2 of the entry distshift](https://dictionaryofml.org/terms/fig/distshift-fig2.svg)

*Figure 2: Distribution shift for a single feature. The solid curve is the probability density function of the probability distribution $\mathbb{P}$ that generated the training set, here a standard normal distribution, with mean $0$ and standard deviation $1$; the dashed curve is the probability density function of the probability distribution $\mathbb{P}'$ at deployment, a normal distribution with mean $2.5$ and standard deviation $1.2$. The dotted threshold of the learned hypothesis $\widehat{h}$ at $x = 1.1$ divides each probability density function into two parts; the area to its right is the fraction of data points whose feature exceeds the threshold, $14\%$ under $\mathbb{P}$ against $88\%$ under $\mathbb{P}'$, both computed from the normal distribution*

Three named special cases of distribution shift correspond to different origins of the shift from $\mathbb{P}$ to $\mathbb{P}'$. These different origins correspond to the factors in the two factorizations of the joint probability distribution. Both hold for any joint probability distribution of a numeric feature vector and label, whether or not a probability density function exists, since the conditional probability distribution is then guaranteed to exist (Billingsley, 1986, Thm. 33.3; Kallenberg, 1997, Thms. 5.3 and 5.4):

$$
\mathbb{P}^{({\mathbf x}, y)}
= \mathbb{P}^{({\mathbf x})} \cdot
\mathbb{P}^{(y \mid {\mathbf x})}
= \mathbb{P}^{(y)} \cdot
\mathbb{P}^{({\mathbf x} \mid y)} \text{.}
$$

Covariate shift and label shift change the marginal distributions $\mathbb{P}^{({\mathbf x})}$ of the feature vector and $\mathbb{P}^{(y)}$ of the label, respectively. Concept drift refers to a change in the conditional probability distribution $\mathbb{P}^{(y \mid {\mathbf x})}$ (with $\mathbb{P}^{({\mathbf x})}$ unchanged) or $\mathbb{P}^{({\mathbf x} \mid y)}$ (with $\mathbb{P}^{(y)}$ unchanged).

The two-station example from the beginning is not a pure case of any of the three. The marginal distribution $\mathbb{P}^{({\mathbf x})}$ changes, since the winter data points have lower feature values. The conditional $\mathbb{P}^{(y \mid {\mathbf x})}$ changes as well: a linear hypothesis learned from the $90$ Nuorgam winter data points has slope $0.77$, against the slope $0.48$ of the hypothesis learned at Helsinki, so a data point whose feature is one degree higher has a label $0.77$ degrees higher in winter, against $0.48$ in summer. The class-conditional $\mathbb{P}^{({\mathbf x} \mid y)}$ changes too. In particular, the two datasets have no label value in common, since every label at Nuorgam in winter is at most $4.6\,^{\circ}$C and every label at Helsinki in summer at least $15.0\,^{\circ}$C.

Distribution shift is detected by comparing summary statistics of the features between the training set and the data points arriving after deployment, or by monitoring the incurred loss whenever labels become available (Quiñonero-Candela et al., 2009).

One mitigation of distribution shift is sample weighting which can be used when the probability distribution $\mathbb{P}$ and $\mathbb{P}'$ are known. Here, with $p$ and $p'$ the pdfs of $\mathbb{P}$ and $\mathbb{P}'$, each data point of the training set is weighted by the ratio $p'({\mathbf x})/p({\mathbf x})$. This weighting corrects covariate shift under two conditions: $\mathbb{P}^{(y \mid {\mathbf x})}$ is unchanged, and every feature vector occurring under $\mathbb{P}'$ also occurs under $\mathbb{P}$, so that the ratio is defined (Quiñonero-Candela et al., 2009). Under these two conditions (Shimodaira, 2000),

$$
p'({\mathbf x}, y) = p(y \mid {\mathbf x})\,
p'({\mathbf x}) =
\big(p'({\mathbf x})/p({\mathbf x})\big)\,
p({\mathbf x}, y) \text{, wherever } p({\mathbf x}) > 0 \text{.}
$$

Thus, the weighted loss has the same expectation under $\mathbb{P}$ as the loss has under $\mathbb{P}'$.

Another approach to mitigate distribution shift is to use online learning, which updates the hypothesis as data points from the changing probability distribution arrive (Quiñonero-Candela et al., 2009). It needs their labels to become available after deployment, which sample weighting does not, and in return it needs neither probability distribution to be known.

**Synonyms:** dataset shift, domain shift.

**See also:** [generalization](https://dictionaryofml.org/terms/generalization.html), [independent and identically distributed assumption](https://dictionaryofml.org/terms/iidasspt.html), [probability distribution](https://dictionaryofml.org/terms/probdist.html), risk, [training set](https://dictionaryofml.org/terms/trainset.html), [test set](https://dictionaryofml.org/terms/testset.html), covariate shift, [transfer learning](https://dictionaryofml.org/terms/transferlearning.html), domain adaptation, sample weighting, [online learning](https://dictionaryofml.org/terms/onlinelearning.html).

## References

- Quiñonero-Candela et al. (2009). Dataset Shift in Machine Learning. *MIT Press*.
- Billingsley (1986). Probability and Measure. *Wiley*.
- Kallenberg (1997). Foundations of Modern Probability. *Springer-Verlag*.
- Shimodaira (2000). Improving predictive inference under covariate shift by weighting the log-likelihood function. *J. Statist. Plann. Inference*. <https://doi.org/10.1016/S0378-3758(00)00115-4>

## Cite this entry

```bibtex
@misc{dictml_distshift,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {distribution shift},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/distshift.html}
}
```


---

# effective dimension

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/effdim.html) · [typeset PDF](https://dictionaryofml.org/terms/effdim.pdf) · [LaTeX](https://dictionaryofml.org/terms/effdim.tex) · [Python demo](https://dictionaryofml.org/terms/effdim.py) · [notebook](https://dictionaryofml.org/terms/effdim.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

The effective dimension of a hypothesis space is its size as seen through the probability distribution of the features: the number of model parameters that make a difference on the data points the distribution produces. For a linear model it is the rank of the second-moment matrix of the feature map, estimated by the rank of the feature matrix, and it never exceeds the number of model parameters. Two hypothesis spaces that differ only outside the support of the distribution have the same effective dimension, and empirical risk minimization on them delivers the same predictions on the data. The same hypothesis space can have different effective dimensions under different data distributions. The Rademacher complexity is the distribution-dependent size measure of learning theory, while the Vapnik–Chervonenkis dimension is its worst case over the whole feature space.

## Definition

Consider the $366$ days of $2024$ at the weather station Krems an der Donau, each a data point with the minimum temperature of the day as its feature $x$ and the maximum temperature as its label $y$. Two hypothesis spaces are candidates for predicting $y$ from $x$: the affine linear model $\mathcal{H}^{(1)} = \{ w_{0} + w_{1} x \}$ with two model parameters, and the linear model $\mathcal{H}^{(2)} =
\{ w_{0} + w_{1} x + w_{2} \max(0, x
- 25) \}$ with three, whose third feature is a rectified linear unit hinge at $25\,^{\circ}$C. The minimum temperature of the record ranges from $-10.1$ to $20.8\,^{\circ}$C, so the hinge is zero on every day. On this record, every hypothesis of $\mathcal{H}^{(2)}$ agrees with a hypothesis of $\mathcal{H}^{(1)}$: the third model parameter changes no prediction that the data can test.

The effective dimension $d_{\rm eff} \left( \mathcal{H} \right)$ of a hypothesis space $\mathcal{H}$ is its size as seen through the probability distribution $\mathbb{P}^{(x)}$ of the features: it counts the model parameters that make a difference on the data points the distribution produces, not the model parameters that are written down. For a linear model with feature map ${\mathbf \Phi}$, the effective dimension is defined as the rank of the second-moment matrix $\mathbb{E}\big\{
{\mathbf \Phi}(x) {\mathbf \Phi}(x)^{\top}
\big\}$. A dataset estimates it by the rank of its feature matrix, which is the trace of the orthogonal projection that least squares applies to the label vector, the effective number of model parameters of a method whose predictions are a linear function of the labels (Hastie et al., 2009, Sect. 7.6). It never exceeds the number of model parameters, and it equals that number only when the features are linearly independent on the support of $\mathbb{P}^{(x)}$. On the Krems record the feature matrix of $\mathcal{H}^{(2)}$ has three columns and rank $2$, since its hinge column is zero, so $d_{\rm eff} \left( \mathcal{H}^{(1)} \right) = d_{\rm eff} \left( \mathcal{H}^{(2)} \right) = 2$. Two hypothesis spaces that differ only outside the support of $\mathbb{P}^{(x)}$ have the same effective dimension.

The consequence for empirical risk minimization is visible in Fig. 1(a). Least squares on $\mathcal{H}^{(1)}$ and on $\mathcal{H}^{(2)}$ delivers the same prediction for every day of the record and the same training error of $19.2$ squared degrees. The hinge coefficient $w_{2}$ is not identifiable: every value gives the same training error, and the two learned hypotheses differ only for a minimum temperature above $25\,^{\circ}$C, a region that no day of the record reaches. There the data say nothing, which is the generalization problem in its sharpest form: a hypothesis is tested only where the features lie.

The effective dimension depends on the probability distribution, not only on the hypothesis space. Fig. 1(b) keeps $\mathcal{H}^{(2)}$ and changes the feature to the maximum temperature of the day, with the maximum temperature of the next day as the label. The maximum temperature exceeds $25\,^{\circ}$C on $97$ of the $365$ day pairs, the hinge column of the feature matrix is no longer zero, its rank is $3$, and least squares on $\mathcal{H}^{(2)}$ now reaches a smaller training error than on $\mathcal{H}^{(1)}$, if only slightly. The same hypothesis space has effective dimension $2$ under one data distribution and $3$ under another.

![Figure 1 of the entry effdim](https://dictionaryofml.org/terms/fig/effdim-fig1.svg)

*Figure 1: The same two hypothesis spaces under two data distributions at Krems an der Donau, $2024$. In (a) the feature is the minimum temperature of the day, which never reaches the hinge at $25\,^{\circ}$C (dotted vertical line): the hypotheses learned by least squares from $\mathcal{H}^{(1)}$ (solid line) and from $\mathcal{H}^{(2)}$ (dashed line, drawn with the hinge coefficient set to $5$) coincide wherever the data lie and differ only beyond it. In (b) the feature is the maximum temperature of the day and the label the maximum temperature of the next day; the hinge is active on the hot days, and the two learned hypotheses differ, slightly, beyond $25\,^{\circ}$C. Open circles are $20$ of the days, every $19$th in date order. Weather data: GeoSphere Austria (GeoSphere Austria, 2026). Data generated by pythondemos/effdim.py*

Beyond linear models, the size measures of learning theory also differ by whether they take a worst case over the feature space or an average over the data distribution. The Vapnik–Chervonenkis dimension is a worst case over all datasets that the feature space admits (Shalev-Shwartz and Ben-David, 2014, Def. 6.5), so it counts the hinge of $\mathcal{H}^{(2)}$ whether or not the data ever reach it: for the classifiers obtained by thresholding a hypothesis at zero, three feature values with one beyond the hinge are shattered by $\mathcal{H}^{(2)}$, since the three values of a hypothesis there are an invertible linear image of $(w_{0}, w_{1}, w_{2})$, while every hypothesis of $\mathcal{H}^{(1)}$ is monotone in $x$ and so realizes no labeling that changes sign twice. The Rademacher complexity is an expectation over datasets drawn from $\mathbb{P}^{(x)}$ and is therefore the effective size of $\mathcal{H}$ for the application at hand (Bartlett and Mendelson, 2002, Def. 2). For a method such as ridge regression, whose learned hypothesis is a linear function of the labels, the trace of that linear map is the effective number of model parameters: it decreases from the rank of the feature matrix toward zero as the regularization parameter $\alpha$ grows (Hastie et al., 2009, Sect. 7.6).

**See also:** [hypothesis space](https://dictionaryofml.org/terms/hypospace.html), model parameter, dimension, linear model, feature map, [feature matrix](https://dictionaryofml.org/terms/featuremtx.html), rank, [probability distribution](https://dictionaryofml.org/terms/probdist.html), [generalization](https://dictionaryofml.org/terms/generalization.html), Vapnik–Chervonenkis dimension, Rademacher complexity, ridge regression.

## References

- GeoSphere Austria (2026). GeoSphere Austria climate station archive. <https://dataset.api.hub.geosphere.at/v1/station/historical/klima-v2-1d>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Bartlett and Mendelson (2002). Rademacher and gaussian complexities: Risk bounds and structural results. *J. Mach. Learn. Res.*. <https://www.jmlr.org/papers/v3/bartlett02a.html>

## Cite this entry

```bibtex
@misc{dictml_effdim,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {effective dimension},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/effdim.html}
}
```


---

# eigenvalue

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-10. [HTML](https://dictionaryofml.org/terms/eigenvalue.html) · [typeset PDF](https://dictionaryofml.org/terms/eigenvalue.pdf) · [LaTeX](https://dictionaryofml.org/terms/eigenvalue.tex) · [Python demo](https://dictionaryofml.org/terms/eigenvalue.py) · [notebook](https://dictionaryofml.org/terms/eigenvalue.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

An eigenvalue of a square matrix ${\mathbf A}$ is a number $\lambda$ for which some nonzero vector ${\mathbf x}$ satisfies ${\mathbf A}{\mathbf x} = \lambda{\mathbf x}$. Such a vector is an eigenvector of ${\mathbf A}$. Multiplying an eigenvector by the matrix rescales it without moving it off the line it spans. The eigenvalues of a diagonalizable matrix form the diagonal factor of its eigenvalue decomposition. An iterative method that applies an affine update, such as gradient descent for linear regression, converges to the unique fixed point of the update, for every initialization, exactly when each eigenvalue of the update matrix has magnitude less than one. The second-smallest eigenvalue of the Laplacian matrix of an undirected graph measures how well the graph is connected and underlies spectral clustering.

## Definition

Whether gradient descent for linear regression converges is decided by the eigenvalues of a single square matrix. A square matrix ${\mathbf A}$ represents a linear map from $\mathbb{R}^{d}$ to $\mathbb{R}^{d}$; the image of a vector ${\mathbf x}$ under this map is the product ${\mathbf A}{\mathbf x}$. Multiplying a vector by a square matrix normally moves it off the line it spans. Some nonzero vectors are exceptions: each stays on the line it spans and is only rescaled. The factor by which such a vector is rescaled is an eigenvalue.

Formally, a number $\lambda \in \mathbb{R}$ is an eigenvalue of a square matrix ${\mathbf A} \in \mathbb{R}^{d \times d}$, one with $d$ rows and $d$ columns, if ${\mathbf A} {\mathbf x} = \lambda {\mathbf x}$ for some nonzero vector ${\mathbf x} \in \mathbb{R}^{d}$; such a vector is an eigenvector of ${\mathbf A}$ (Strang, 2016, Sect. 6.1). For an eigenvector, the image ${\mathbf A}{\mathbf x}$ stays on the line ${\mathbf x}$ spans, pointing the same way when $\lambda > 0$ and the opposite way when $\lambda < 0$ (see Fig. 1). Arranging $d$ linearly independent eigenvectors of a diagonalizable matrix as the columns of $\mathbf{V}$, and the corresponding eigenvalues in a diagonal matrix ${\boldsymbol \Lambda}$, yields the eigenvalue decomposition ${\mathbf A} = \mathbf{V} {\boldsymbol \Lambda} \mathbf{V}^{-1}$ (Strang, 2016, Sect. 6.2).

![Figure 1 of the entry eigenvalue](https://dictionaryofml.org/terms/fig/eigenvalue-fig1.svg)

*Figure 1: An eigenvector ${\mathbf x}$ of ${\mathbf A}$ (solid arrow) and its image ${\mathbf A} {\mathbf x} = \lambda {\mathbf x}$ (dashed arrow), here with eigenvalue $\lambda = 1.8$. The image stays on the dotted line that ${\mathbf x}$ spans, rescaled by $\lambda$. A negative $\lambda$ would point the image the other way along the same line*

Eigenvalues can be used to study iterative machine learning methods with an update of the form

$$
{\mathbf w}^{(t+1)} = {\mathbf M} {\mathbf w}^{(t)} + {\mathbf b}
\text{, for } t = 0,1,\ldots \text{.}
$$

An example is gradient descent for linear regression, whose update operator $\mathcal{F}^{(\eta)} : {\mathbf w} \mapsto {\mathbf M} {\mathbf w} + {\mathbf b}$ is affine with iteration matrix ${\mathbf M} = \mathbf{I} - (2\eta/m) {\mathbf X}^{\top}{\mathbf X}$. Here, $\mathbf{I}$ is the identity matrix, $\eta$ the learning rate, $m$ the number of data points, and ${\mathbf X}$ the feature matrix of the training set (see linear regression). Such an iteration converges to the unique fixed point $\widehat{{\mathbf w}}$ of $\mathcal{F}^{(\eta)}$, for every initialization, if and only if the spectral radius $\rho$ of ${\mathbf M}$, the largest magnitude of an eigenvalue of ${\mathbf M}$, satisfies $\rho < 1$ (Horn and Johnson, 2013, Thm. 5.6.12). For a diagonalizable ${\mathbf M}$, the eigenvalue decomposition shows why: in the basis of its eigenvectors, each iteration rescales the components of the error ${\mathbf w}^{(t)} - \widehat{{\mathbf w}}$ by the eigenvalues, so the error shrinks exactly when every eigenvalue has magnitude below one.

Beyond the convergence of iterative methods, eigenvalues are also used to measure the connectivity of an undirected graph $\mathcal{G}$ with the number of nodes $n$. The Laplacian matrix ${\mathbf L}^{(\mathcal{G})}$ of $\mathcal{G}$ is positive semi-definite, with eigenvalues $0 = \lambda_{1} \leq \lambda_{2} \leq \ldots \leq \lambda_{n}$ (Luxburg, 2007, Prop. 1). The second-smallest eigenvalue $\lambda_{2}$, the algebraic connectivity of $\mathcal{G}$, quantifies connectivity: $\lambda_{2} > 0$ if and only if $\mathcal{G}$ is connected (Luxburg, 2007, Prop. 2), and $\lambda_{2}$ close to zero indicates two clusters of nodes with few edges between them (Chung, 1997; Fiedler, 1973; Luxburg, 2007, Sect. 5.1). Fig. 2 shows three edge sets on the same six nodes: adding edges increases $\lambda_{2}$ from $0$, for two clusters without a connecting edge, to $6$, for the complete graph. The value $6$ is the largest possible. For a graph with unit edge weights, the Laplacian matrix of $\mathcal{G}$ and that of the complement $\bar{\mathcal{G}}$, which has the edges $\mathcal{G}$ lacks, add to $n \mathbf{I} - \mathbf{1} \mathbf{1}^{\top}$. A unit-norm eigenvector ${\mathbf x}$ of a positive eigenvalue is orthogonal to the constant vector $\mathbf{1}$, an eigenvector of the eigenvalue $0$, so ${\mathbf x}^{\top} {\mathbf L}^{(\mathcal{G})} {\mathbf x} = n - {\mathbf x}^{\top} {\mathbf L}^{(\bar{\mathcal{G}})} {\mathbf x} \leq n$, since ${\mathbf L}^{(\bar{\mathcal{G}})}$ is positive semi-definite. For the complete graph the complement has no edges, so ${\mathbf L}^{(\mathcal{G})} = n \mathbf{I} - \mathbf{1} \mathbf{1}^{\top}$ and every eigenvalue other than $\lambda_{1} = 0$ equals $n$.

![Figure 2 of the entry eigenvalue](https://dictionaryofml.org/terms/fig/eigenvalue-fig2.svg)

*Figure 2: Three edge sets on the same six nodes, with increasing connectivity from left to right, and the second-smallest eigenvalue $\lambda_{2}$ of the corresponding Laplacian matrix. (a) Two clusters without a connecting edge, so the graph is disconnected and $\lambda_{2} = 0$. (b) One edge connects the clusters, so the graph is connected and $\lambda_{2} \approx 0.44$ is positive but close to zero. (c) The complete graph, where every pair of nodes is connected, so the graph is connected and $\lambda_{2} = 6$ attains the largest possible value, the number of nodes $n = 6$, since no eigenvalue of the Laplacian matrix of a graph with $n$ nodes and unit edge weights exceeds $n$. Data generated by pythondemos/eigenvalue.py*

The eigenvector corresponding to $\lambda_{2}$, referred to as the Fiedler vector, can be used to partition $\mathcal{G}$ into two clusters: each node is assigned according to the sign of its own component of the Fiedler vector (Fiedler, 1975; Spielman, 2025).

**See also:** [matrix](https://dictionaryofml.org/terms/matrix.html), eigenvector, [eigenvalue decomposition](https://dictionaryofml.org/terms/evd.html), spectral radius, [gradient descent](https://dictionaryofml.org/terms/gd.html), [linear regression](https://dictionaryofml.org/terms/linreg.html), contractive operator, fixed-point iteration, Laplacian matrix, algebraic connectivity, Fiedler vector, spectral clustering.

## References

- Strang (2016). Introduction to Linear Algebra. *Wellesley-Cambridge Press*.
- Horn and Johnson (2013). Matrix Analysis. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781139020411>
- Luxburg (2007). A tutorial on spectral clustering. *Statist. Comput.*. <https://doi.org/10.1007/s11222-007-9033-z>
- Chung (1997). Spectral Graph Theory. *American Mathematical Society*.
- Fiedler (1973). Algebraic connectivity of graphs. *Czechoslovak Mathematical Journal*.
- Fiedler (1975). A property of eigenvectors of nonnegative symmetric matrices and its application to graph theory. *Czechoslovak Mathematical Journal*.
- Spielman (2025). Spectral and Algebraic Graph Theory (Incomplete Draft). *Ebook*. <http://cs-www.cs.yale.edu/homes/spielman/sagt>

## Cite this entry

```bibtex
@misc{dictml_eigenvalue,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {eigenvalue},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/eigenvalue.html}
}
```


---

# expectation–maximization (EM)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/em.html) · [typeset PDF](https://dictionaryofml.org/terms/em.pdf) · [LaTeX](https://dictionaryofml.org/terms/em.tex) · [Python demo](https://dictionaryofml.org/terms/em.py) · [notebook](https://dictionaryofml.org/terms/em.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

The expectation–maximization (EM) algorithm is an iterative optimization method for a particular class of maximum likelihood problems. In particular, EM is useful for probabilistic models that involve observed and unobserved (or latent) attributes of a data point. Each iteration of the EM algorithm alternates an E-step with an M-step. The E-step computes the posterior distribution of the unobserved attributes under the current model parameters. The M-step minimizes the negative log-likelihood of the complete data points averaged under this posterior distribution, which for many probabilistic models has a closed form. The EM algorithm is an instance of a majorize–minimize method. A canonical soft clustering method uses the EM algorithm to learn the model parameters of a Gaussian mixture model.

## Definition

The nightly minimum temperatures recorded over one year at the weather station of Krems an der Donau (Austria) scatter around two regimes, cold-season and warm-season nights. A probabilistic model for such data points has, besides the observed temperature, an unobserved attribute: the regime a night belongs to. Fig. 1 shows the histogram of the temperatures together with a Gaussian mixture model consisting of a separate normal distribution for each regime. The EM algorithm (Dempster et al., 1977) is an iterative optimization method for approximately solving maximum likelihood optimization problems for probabilistic models with an unobserved attribute (Bishop, 2006, Sect. 9.4; Hastie et al., 2009, Sect. 8.5; Murphy, 2012, Sect. 11.4.7).

![Figure 1 of the entry em](https://dictionaryofml.org/terms/fig/em-fig1.svg)

*Figure 1: Histogram of the $366$ nightly minimum temperatures of 2024 at the weather station Krems an der Donau, with the two-component Gaussian mixture model learned by EM (solid curve) and its two weighted components, the cold-season component (dashed curve) and the warm-season component (dotted curve). Each night belongs to one of the two regimes, but which one is not observed. Data generated by pythondemos/em.py*

Consider a dataset $\mathcal{D} = \{{\mathbf x}^{(1)}, \ldots,
{\mathbf x}^{(m)}\}$ whose data points are realizations of independent and identically distributed RVs. The underlying probability distribution is a probabilistic model $p^{({\mathbf x}',y')}\left({\mathbf x},y;{\mathbf w}\right)$ for the observed feature vector ${\mathbf x}$ and an unobserved attribute $y$ with values in a finite set $\mathcal{Y}$. This probabilistic model is a family of probability distributions parameterized by model parameters ${\mathbf w}$ in a parameter space $\mathcal{W}$. Maximum likelihood is an instance of empirical risk minimization: it minimizes the empirical risk

\begin{equation}
\tag{1}
\widehat{L}\big({\mathbf w}|\mathcal{D}\big) :=
- \frac{1}{m} \sum_{r=1}^{m}
\ln \sum_{y \in \mathcal{Y}}
p^{({\mathbf x}',y')}\left({\mathbf x}^{(r)},y;{\mathbf w}\right)\text{,}
\end{equation}

the average negative log-likelihood of the observed feature vectors. The sum inside the logarithm is what makes Eq. (1) hard to minimize directly: if the attributes $y^{(r)}$ were observed, the empirical risk would be an average of $-\ln p^{({\mathbf x}',y')}\left({\mathbf x}^{(r)},y^{(r)};{\mathbf w}\right)$, which for many probabilistic models, among them the Gaussian mixture model, has a closed-form minimizer (Bishop, 2006, Sect. 9.3.1).

The EM algorithm solves the maximum likelihood problem Eq. (1) by alternating two steps, which together form one iteration. The E-step computes, for every data point, the posterior distribution of the unobserved attribute under the current model parameters $\widehat{{\mathbf w}}$,

\begin{equation}
\tag{2}
p^{(y'|{\mathbf x}')}\left(y;\widehat{{\mathbf w}}\right)
= \frac{p^{({\mathbf x}',y')}\left({\mathbf x}^{(r)},y;\widehat{{\mathbf w}}\right)}
{\sum_{y'' \in \mathcal{Y}}
p^{({\mathbf x}',y')}\left({\mathbf x}^{(r)},y'';\widehat{{\mathbf w}}\right)}
\quad \text{for } y \in \mathcal{Y}
\text{ and } r = 1, \ldots, m\text{.}
\end{equation}

These posterior distributions fix the surrogate that the E-step of Algorithm 1 forms, the empirical risk of the complete data points averaged under them,

\begin{equation}
\tag{3}
Q({\mathbf w} \mid \widehat{{\mathbf w}}) :=
- \frac{1}{m} \sum_{r=1}^{m}
\sum_{y \in \mathcal{Y}}
p^{(y'|{\mathbf x}')}\left(y;\widehat{{\mathbf w}}\right)
\ln \frac{p^{({\mathbf x}',y')}\left({\mathbf x}^{(r)},y;{\mathbf w}\right)}
{p^{(y'|{\mathbf x}')}\left(y;\widehat{{\mathbf w}}\right)}\text{.}
\end{equation}

The M-step minimizes this surrogate over ${\mathbf w} \in
\mathcal{W}$. Algorithm 1 states the method in full.

![Algorithm 1 of the entry em](https://dictionaryofml.org/terms/fig/em-alg1.svg)

*Algorithm 1: Expectation-maximization*

The EM algorithm is a majorize–minimize method: each iteration minimizes a surrogate $Q({\mathbf w} \mid \widehat{{\mathbf w}})$ of the empirical risk Eq. (1). The surrogate Eq. (3) has three properties (Bishop, 2006, Sect. 9.4, whose lower bound on the log-likelihood is this surrogate written for the negative log-likelihood; Hunter and Lange, 2004, Sect. 2). First, it majorizes the empirical risk, $Q({\mathbf w} \mid \widehat{{\mathbf w}}) \geq
\widehat{L}\big({\mathbf w}|\mathcal{D}\big)$ for all ${\mathbf w} \in \mathcal{W}$, by Jensen's inequality applied to the concave logarithm. Second, it is tight at the current iterate, $Q(\widehat{{\mathbf w}} \mid \widehat{{\mathbf w}})
= \widehat{L}\big(\widehat{{\mathbf w}}|\mathcal{D}\big)$. Third, it is minimized exactly by the M-step.

Any model parameters ${\mathbf w}^{(t+1)}$ that minimize the surrogate built at ${\mathbf w}^{(t)}$ therefore satisfy $\widehat{L}\big({\mathbf w}^{(t+1)}|\mathcal{D}\big) \leq
Q({\mathbf w}^{(t+1)} \mid {\mathbf w}^{(t)}) \leq
Q({\mathbf w}^{(t)} \mid {\mathbf w}^{(t)}) =
\widehat{L}\big({\mathbf w}^{(t)}|\mathcal{D}\big)$: no iteration increases the empirical risk (Dempster et al., 1977, Thm. 1).

The monotone decrease of the empirical risk Eq. (1) along the EM iterates yields a stopping criterion: the iteration stops as soon as the decrease of the empirical risk achieved by the last iteration falls below a given threshold (Bishop, 2006, Sect. 9.2.2). The empirical risk Eq. (1) is in general a non-convex function of the model parameters, and EM might converge to a local minimum (Bishop, 2006, Sect. 9.2.2). It is therefore typically run several times with different initializations, keeping the run with the smallest empirical risk (Murphy, 2012, Sect. 11.3.2).

The EM algorithm is a fixed-point iteration $\widehat{{\mathbf w}}^{(t+1)} =
\mathcal{F}\big(\widehat{{\mathbf w}}^{(t)}\big)$. The operator $\mathcal{F}$ maps the current model parameters $\widehat{{\mathbf w}}$ to the result of one E-step followed by one M-step. The E-step Eq. (2) at $\widehat{{\mathbf w}}$ builds the surrogate $Q(\cdot \mid \widehat{{\mathbf w}})$, and the M-step Eq. (3) minimizes it,

$$
\mathcal{F}(\widehat{{\mathbf w}}) :=
\mathop{\mathrm{arg\,min}}_{{\mathbf w} \in \mathcal{W}}
Q({\mathbf w} \mid \widehat{{\mathbf w}})\text{.}
$$

The fixed points of $\mathcal{F}$ are model parameters that minimize their own surrogate: there, the surrogate touches the empirical risk from above, so no further descent step is available. If the empirical risk and the surrogate are differentiable and the fixed point $\widehat{{\mathbf w}}$ lies in the interior of $\mathcal{W}$, the gradient of the empirical risk vanishes at $\widehat{{\mathbf w}}$. Indeed, the difference $Q({\mathbf w} \mid \widehat{{\mathbf w}}) -
\widehat{L}\big({\mathbf w}|\mathcal{D}\big)$ is nonnegative for all ${\mathbf w}$ and zero at ${\mathbf w} = \widehat{{\mathbf w}}$, so its gradient vanishes there, and the gradient of $Q(\cdot \mid
\widehat{{\mathbf w}})$ vanishes at its interior minimum $\widehat{{\mathbf w}}$. Under the conditions of Wu (1983, Thm. 2), every limit point of the EM iterates is such a stationary point of the empirical risk.

EM is used to learn the model parameters of a Gaussian mixture model, where the unobserved attribute of a data point is its cluster index $I^{(r)} \in \{1, \ldots,
k\}$ and the model parameters are the cluster probabilities $p_{c}$, means ${\boldsymbol \mu}^{(c)}$, and covariance matrices $\mathbf{C}^{(c)}$ (Bishop, 2006, Sect. 9.2.2). The E-step Eq. (2) becomes the degree of belonging

\begin{equation}
\tag{4}
u_{c}^{(r)} :=
\mathbb{P}\left(I^{(r)} = c \mid {\mathbf x}^{(r)}\right) =
\frac{p_{c}\,
\mathcal{N}\left({\boldsymbol \mu}^{(c)},\mathbf{C}^{(c)}\right)({\mathbf x}^{(r)})}
{\sum_{c'=1}^{k} p_{c'}\,
\mathcal{N}\left({\boldsymbol \mu}^{(c')},\mathbf{C}^{(c')}\right)({\mathbf x}^{(r)})}\text{,}
\end{equation}

and minimizing the surrogate Eq. (3) yields the M-step in closed form. The cluster probability $p_{c}$ is the average of $u_{c}^{(r)}$ over the data points, while ${\boldsymbol \mu}^{(c)}$ and $\mathbf{C}^{(c)}$ are the mean and covariance matrix of the feature vectors weighted by $u_{c}^{(r)}$. The learned degrees of belonging Eq. (4) constitute a soft clustering. Fixing every cluster probability at $1/k$ and every covariance matrix at $\sigma^{2} \mathbf{I}$, a multiple of the identity matrix $\mathbf{I}$ with a common $\sigma^{2}$ that shrinks towards $0$ turns the degrees of belonging into $0$ or $1$, and EM reduces to Lloyd's algorithm for k-means (Bishop, 2006, Sect. 9.3.2).

Consider next, as an illustration of the EM algorithm, the Gaussian mixture model of Fig. 1 for the temperature dataset. The feature of a data point is the nightly minimum temperature $x \in \mathbb{R}$, and its unobserved attribute is the regime $y \in \{\mathrm{cold}, \mathrm{warm}\}$. The model parameters are the probabilities $p_{\mathrm{cold}}$, $p_{\mathrm{warm}} = 1 - p_{\mathrm{cold}}$ of the two regimes, the means $\mu_{\mathrm{cold}}$, $\mu_{\mathrm{warm}}$ and the variances $\sigma^{2}_{\mathrm{cold}}$, $\sigma^{2}_{\mathrm{warm}}$ of the two normal distributions, so ${\mathbf w} = (p_{\mathrm{cold}}, p_{\mathrm{warm}},
\mu_{\mathrm{cold}}, \mu_{\mathrm{warm}},
\sigma^{2}_{\mathrm{cold}}, \sigma^{2}_{\mathrm{warm}})^{\top}
\in \mathbb{R}^{6}$. The E-step Eq. (2) computes for every night the probability of the cold regime, and the M-step Eq. (3) recomputes ${\mathbf w}$ as averages weighted by these probabilities. On the $366$ nights of 2024, EM learns $p_{\mathrm{cold}} = 0.83$, $\mu_{\mathrm{cold}} = 5.1$, $\mu_{\mathrm{warm}} = 17.2$, $\sigma^{2}_{\mathrm{cold}} = 39.7$, and $\sigma^{2}_{\mathrm{warm}} = 3.3$.

Fig. 2 shows the majorize–minimize view for this Gaussian mixture model as the cold-season mean $\mu_{\mathrm{cold}}$ varies, while $p_{\mathrm{cold}}$, $\mu_{\mathrm{warm}}$ and $\sigma^{2}_{\mathrm{warm}}$ are fixed at their learned values and $\sigma^{2}_{\mathrm{cold}}$ is fixed at $16$. The empirical risk is then a function of $\mu_{\mathrm{cold}}$ alone, and so is each surrogate. Each surrogate lies above the empirical risk, touches it at its iterate, and its minimizer is the next iterate. Fig. 3 shows how the empirical risk decreases monotonically as the EM iterations proceed.

![Figure 2 of the entry em](https://dictionaryofml.org/terms/fig/em-fig2.svg)

*Figure 2: The majorize–minimize view of EM for the two-component Gaussian mixture model of Fig. 1 with a single free model parameter, the cold-season mean $\mu_{\mathrm{cold}}$. The other model parameters are fixed: $p_{\mathrm{cold}}$, $\mu_{\mathrm{warm}}$, and $\sigma^{2}_{\mathrm{warm}}$ at their learned values, and $\sigma^{2}_{\mathrm{cold}} = 16$, smaller than its learned value so that the surrogates differ visibly from the empirical risk. Solid: the empirical risk as a function of $\mu_{\mathrm{cold}}$. Dashed, dotted and dash-dotted: the surrogates built by the E-step at the first three iterates ${\mathbf w}^{(0)}, {\mathbf w}^{(1)}, {\mathbf w}^{(2)}$ of EM on $\mu_{\mathrm{cold}}$, starting from $\mu_{\mathrm{cold}} = -8$. Each surrogate lies above the empirical risk, touches it at its iterate, and is minimized at the next iterate. Data generated by pythondemos/em.py*

![Figure 3 of the entry em](https://dictionaryofml.org/terms/fig/em-fig3.svg)

*Figure 3: EM learning all six model parameters of the two-component Gaussian mixture model of Fig. 1 on the $366$ nightly minimum temperatures of 2024 at Krems: the negative log-likelihood decreases monotonically as the iterations proceed and settles at a fixed point of the EM operator $\mathcal{F}$. Data generated by pythondemos/em.py*

**See also:** maximum likelihood, probabilistic model, [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), majorize–minimize, fixed-point iteration, fixed point, posterior distribution, [Gaussian mixture model](https://dictionaryofml.org/terms/gmm.html), soft clustering, degree of belonging, [$k$-means](https://dictionaryofml.org/terms/kmeans.html), Lloyd's algorithm, optimization problem.

## References

- Dempster et al. (1977). Maximum likelihood from incomplete data via the EM algorithm. *J. Roy. Statist. Soc.: Ser. B (Methodological)*. <https://doi.org/10.1111/j.2517-6161.1977.tb01600.x>
- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Murphy (2012). Machine Learning: A Probabilistic Perspective. *MIT Press*.
- Hunter and Lange (2004). A Tutorial on MM Algorithms. *Amer. Statistician*. <https://doi.org/10.1198/0003130042836>
- Wu (1983). On the convergence properties of the EM algorithm. *Ann. Statist.*. <https://doi.org/10.1214/aos/1176346060>

## Cite this entry

```bibtex
@misc{dictml_em,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {expectation–maximization (EM)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/em.html}
}
```


---

# ensemble

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-07. [HTML](https://dictionaryofml.org/terms/ensemble.html) · [typeset PDF](https://dictionaryofml.org/terms/ensemble.pdf) · [LaTeX](https://dictionaryofml.org/terms/ensemble.tex) · [Python demo](https://dictionaryofml.org/terms/ensemble.py) · [notebook](https://dictionaryofml.org/terms/ensemble.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

An ensemble method combines several machine learning methods, each referred to as a base learner. Ensemble methods differ in how they construct the base learners and in how they combine their results. Bagging trains each base learner on a resampled copy of the training set. Boosting runs them sequentially, each one fitted to the errors of the previous ones. Stacking uses base learners to train different models on the same training set and learns an aggregation rule. For the squared error loss the gain is exact: the error of the average prediction is the average error of the base learners minus their average disagreement, so base learners that disagree are what makes an ensemble worth forming.

## Definition

Predicting the maximum temperature of a day at the weather station Krems an der Donau from the minimum temperature of the same day can be done with a linear model, with a decision tree, or with an artificial neural network. Fig. 1 shows all three learned from the $366$ days of $2020$. None of them is the best everywhere: the linear hypothesis misses the curvature at the cold end, the decision tree answers with a staircase, and the artificial neural network bends where the other two cannot. Model selection would keep one of them and discard the rest. Combining these base learners can do better than choosing one of them.

![Figure 1 of the entry ensemble](https://dictionaryofml.org/terms/fig/ensemble-fig1.svg)

*Figure 1: Three base learners on one task: the maximum temperature of a day at Krems an der Donau against the minimum temperature of the same day. The dots are $20$ of the $366$ days of $2020$, every $19$th day. The linear model (dash-dotted curve), the decision tree (dotted curve) and the artificial neural network (dashed curve) are each learned from the same $244$ training days. On the $122$ held-back days their average squared errors are $26.20$, $27.09$, and $26.13$, so none of them is the best; the average of the three (solid curve) reaches $25.95$. Data generated by pythondemos/ensemble.py*

Combining several machine learning methods into one predictor is an ensemble method: each combined method is a base learner (Hastie et al., 2009, Ch. 16). The base learners can be empirical risk minimization-based, using different choices for the loss, model, and training set. Of $M$ base learners, base learner $\ell$ learns a hypothesis $\widehat{h}^{(\ell)}$ from its hypothesis space $\mathcal{H}^{(\ell)}$ and its, possibly perturbed, training set $\widetilde{\mathcal{D}}^{(\ell)}$, and an aggregation rule $\phi_{\text{agg}}$ combines the outputs of these hypotheses into the final prediction. The aggregation can amount to averaging (in regression), as in Fig. 1, or to a majority vote (in classification); Fig. 2 shows the general shape.

![Figure 2 of the entry ensemble](https://dictionaryofml.org/terms/fig/ensemble-fig2.svg)

*Figure 2: Generic ensemble with three base learners: base learner $\ell \in \{1, 2, 3\}$ uses empirical risk minimization to learn $\widehat{h}^{(\ell)} \in \mathcal{H}^{(\ell)}$ from the perturbed training set $\widetilde{\mathcal{D}}^{(\ell)}$. A base learner might also use the output of other base learners (dashed arrows). The final hypothesis $\widehat{h}$ is obtained by aggregating the hypotheses of the base learners via an aggregation rule $\phi_{\text{agg}}$*

What the combination gains is exactly the disagreement among the base learners. For the squared error loss, the error of the average prediction is the average error of the base learners minus their average disagreement, where the disagreement of a base learner is the squared difference between its own prediction and the average one (Krogh and Vedelsby, 1994). This is an identity, not an approximation: on the held-back days of Fig. 1 the average member error $26.47$ minus the average disagreement $0.52$ is the $25.95$ that the average prediction actually incurs. Two consequences follow. The average is never worse than its base learners are on average, since a disagreement cannot be negative. And base learners that agree everywhere gain nothing, which is why they are trained on different models or on different data points. Outperforming the *best* single base learner is a stronger claim, and one the identity does not guarantee: over $50$ repetitions on random halves of the training set, the average did so $22$ times.

The same point in terms of variance: consider $M$ base learners whose predictions are identically distributed, each with variance $\sigma^{2}$ and pairwise correlation $\rho$. The average of these predictions has variance (Hastie et al., 2009, Sect. 15.2)

$$
\rho \sigma^{2} + \frac{1-\rho}{M}\,\sigma^{2}\text{.}
$$

The second of these two summands vanishes as $M$ grows, while the first, set by the correlation between the base learners, limits what averaging can achieve.

Ensemble methods differ in how they construct the base learners and in how they combine their results. Bagging trains each base learner on a random resample of the training set and averages them; it reduces the variance of an unstable base learner such as a deep decision tree, while the systematic part of its error, the bias, is left unchanged (Hastie et al., 2009, Sect. 8.7). A random forest is bagging of decision trees with, in addition, a random choice of the features considered at each split, which lowers the correlation between the trees (Breiman, 2001, Sect. 3; Hastie et al., 2009, Sect. 15.2).

Boosting runs the base learners sequentially, each one fitted to the errors of the previous ones. The number of rounds is a hyperparameter: the training error keeps decreasing while the validation error eventually rises, so boosting overfits when run for too long (Hastie et al., 2009, Sect. 10.12). Stacking trains the base learners on the same training set with different models and learns the aggregation rule itself by a second machine learning method. The second method is trained on predictions for data points that did not belong to the training sets of the base learners. Otherwise the base learners that fit the training set best are favored regardless of how they generalize (Hastie et al., 2009, Sect. 8.8).

**See also:** [bagging](https://dictionaryofml.org/terms/bagging.html), [boosting](https://dictionaryofml.org/terms/boosting.html), stacking, [base learner](https://dictionaryofml.org/terms/baselearner.html), [random forest](https://dictionaryofml.org/terms/randomforest.html), [variance](https://dictionaryofml.org/terms/variance.html), [correlation](https://dictionaryofml.org/terms/correlation.html).

## References

- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Krogh and Vedelsby (1994). Neural Network Ensembles, Cross Validation, and Active Learning. *Adv. Neural Inf. Process. Syst. (NeurIPS)*.
- Breiman (2001). Random Forests. *Machine Learning*. <https://doi.org/10.1023/A:1010933404324>

## Cite this entry

```bibtex
@misc{dictml_ensemble,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {ensemble},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/ensemble.html}
}
```


---

# empirical risk minimization (ERM)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/erm.html) · [typeset PDF](https://dictionaryofml.org/terms/erm.pdf) · [LaTeX](https://dictionaryofml.org/terms/erm.tex) · [Python demo](https://dictionaryofml.org/terms/erm.py) · [notebook](https://dictionaryofml.org/terms/erm.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Empirical risk minimization (ERM) selects a hypothesis by minimizing the average loss on a given training set $\mathcal{D}^{(\rm train)}$, the empirical risk. Machine learning methods aim at a hypothesis that incurs a small loss on any data point, and ERM replaces this unknown quantity by its average over the training set. Different machine learning methods arise from different choices of the model $\mathcal{H}$ and the loss function $L$, both of which depend on the feature space $\mathcal{X}$ and the label space $\mathcal{Y}$ of a given method.

## Definition

A weather forecaster predicting tomorrow's maximum daytime temperature is useful only if its predictions are accurate on the large majority of days, not only on the days it was trained on. This forecaster is a hypothesis: it maps a day's features to a prediction of tomorrow's temperature, and its loss quantifies how far that prediction lies from the measured value. More generally, machine learning methods aim to find a hypothesis that incurs a small loss for any data point. The phrase *any data point* is made precise by a probabilistic model for the generation of data points, as follows.

If data points are assumed to be drawn from a probability distribution, then the risk of a hypothesis is defined as the expectation of its loss under that probability distribution. The ideal choice is then the hypothesis with minimal risk. However, in most machine learning applications, the underlying probability distribution is not known. Instead, only a finite training set $\mathcal{D}^{(\rm train)}$ of data points is available. ERM replaces the unknown risk with its sample-average surrogate, the empirical risk on $\mathcal{D}^{(\rm train)}$, and picks the hypothesis that minimizes this surrogate (Hastie et al., 2009; Murphy, 2012; Shalev-Shwartz and Ben-David, 2014; Vapnik, 2000).

For a fixed choice of loss function and hypothesis space, ERM is a map $\mathcal{A}$ that reads in a training set $\mathcal{D}^{(\rm train)} = \{{\mathbf z}^{(r)}\}_{r=1}^{m}$ of data points ${\mathbf z}^{(r)} = \left( {\mathbf x}^{(r)},y^{(r)} \right)$. The map returns a learned hypothesis $\widehat{h} = \mathcal{A}(\mathcal{D}^{(\rm train)}) \in \mathcal{H}$ that minimizes the empirical risk on $\mathcal{D}^{(\rm train)}$,

$$
\widehat{h} \in \mathop{\mathrm{arg\,min}}_{h \in \mathcal{H}}
\frac{1}{m}
\sum_{r=1}^{m}
L\left({\mathbf z}^{(r)},h \right) \text{.}
$$

The hypothesis $\widehat{h}$ is chosen from the underlying hypothesis space (or model) $\mathcal{H}$. For an ERM-based method, model training is to compute $\mathcal{A}$.

Different machine learning methods arise from different design choices for the two components of the map $\mathcal{A}$: the model $\mathcal{H}$ and the loss function $L$. Both depend on the nature of the data points, i.e., on the feature space $\mathcal{X}$ and the label space $\mathcal{Y}$. For example, image data points with a high-dimensional feature space and a finite label space are typically handled by an artificial neural network together with the logistic loss (Goodfellow et al., 2016, Ch. 9). Conversely, data points with few real-valued features and a real-valued label often allow for a linear model together with the squared error loss (Hastie et al., 2009, Ch. 3).

Fig. 1 illustrates ERM for a linear model on data points with a single feature $x$ and label $y$. Each hypothesis is a linear map $h(x) = w_{1} x + w_{0}$ with model parameters $w_{0}, w_{1}$. ERM picks the model parameters that minimize the empirical risk on $\mathcal{D}^{(\rm train)}$.

![Figure 1 of the entry erm](https://dictionaryofml.org/terms/fig/erm-fig1.svg)

*Figure 1: ERM learns a hypothesis $\widehat{h} \in \mathcal{H}$ out of a model $\mathcal{H}$ by minimizing the empirical risk (or average loss) $\tfrac{1}{m} \sum_{r=1}^{m} L\left({\mathbf z}^{(r)},h \right)$ over candidate hypotheses $h \in \mathcal{H}$ on a training set $\mathcal{D}^{(\rm train)}$. The arrows indicate the prediction errors incurred by the learned hypothesis $\widehat{h}$ (dashed line) for each data point in the training set (filled circles); the axes carry the feature $x$ and the label $y$*

In practice, the map $\mathcal{A}$ is computed by an iterative optimization method. Many of these optimization methods can be represented as a fixed-point iteration that starts with an initial hypothesis $h^{(0)}$ and then repeatedly applies an update operator $\mathcal{F}$ (Bauschke and Combettes, 2011, Sect. 5.2, Cor. 27.9),

$$
h^{(t+1)} = \mathcal{F}\big( h^{(t)} \big) \text{, for } t=0,1,\ldots \text{.}
$$

The update operator $\mathcal{F}$ depends on the training set, the loss function, and the hypothesis space. It is chosen such that its fixed points are the stationary points of the empirical risk. These are its minimizers when the empirical risk is a convex function of the hypothesis (Boyd and Vandenberghe, 2004, Sect. 4.2.3), and need not be otherwise, e.g., for an artificial neural network.

For ERM using a parametric model $\mathcal{H}$, the fixed-point iteration can be formulated directly in terms of the model parameters ${\mathbf w}$ instead of the hypothesis $h$. In particular, starting with initial model parameters ${\mathbf w}^{(0)}$, the update operator $\mathcal{F}$ is applied repeatedly to the model parameters,

$$
{\mathbf w}^{(t+1)} = \mathcal{F}\big( {\mathbf w}^{(t)} \big) \text{, for } t=0,1,\ldots \text{.}
$$

One example of such a fixed-point iteration is gradient descent, which updates the model parameters in the direction of the negative gradient of the empirical risk (which needs to be differentiable).

Two requirements delimit where ERM applies: full feedback on the loss and access to the whole training set at once. First, ERM presupposes that the loss $L\left({\mathbf z}^{(r)},h \right)$ can be evaluated for every hypothesis $h \in \mathcal{H}$ and every training data point ${\mathbf z}^{(r)}$. This full-feedback requirement is what distinguishes ERM from reinforcement learning algorithms, which use partial feedback. In particular, at each time instant $t = 1, 2, \ldots$, a reinforcement learning algorithm chooses an action, the prediction of its current hypothesis $\widehat{h}^{(t)}$, and observes only the loss incurred by that action. It has no information about the loss that would have been incurred by any other hypothesis $h \in \mathcal{H} \setminus
\{\widehat{h}^{(t)}\}$.

Second, ERM uses the whole training set at once. This requirement can be relaxed: ERM can also be implemented as an online algorithm, which is useful when the data points in $\mathcal{D}^{(\rm train)}$ can only be accessed sequentially. Such sequential access arises, for instance, when memory limits prevent loading the entire training set at once. The canonical online form of ERM is the Follow-The-Leader algorithm, which, at every round, solves a partial ERM problem on all data points seen so far. Assuming that the data point ${\mathbf z}^{(t)}$ arrives at time instant $t=1,2,\ldots$, Follow-The-Leader updates the learned hypothesis $\widehat{h}^{(t)}$ by

$$
\widehat{h}^{(t)} \in \mathop{\mathrm{arg\,min}}_{h \in \mathcal{H}}
\frac{1}{t} \sum_{r=1}^{t}
L\left({\mathbf z}^{(r)},h \right) \text{.}
$$

Implementing this partial ERM separately for each time instant can become computationally expensive (and wasteful), as it does not exploit the similarity between consecutive partial ERM problems. This similarity can be exploited by online gradient descent, which performs a single gradient descent step on the new data point ${\mathbf z}^{(t)}$ at every time instant $t$ (Hazan, 2022, Sect. 3.1).

**Synonyms:** empirical loss minimization.

**See also:** empirical risk, risk, [loss](https://dictionaryofml.org/terms/loss.html), [training set](https://dictionaryofml.org/terms/trainset.html), [hypothesis space](https://dictionaryofml.org/terms/hypospace.html), [hypothesis](https://dictionaryofml.org/terms/hypothesis.html), [gradient descent](https://dictionaryofml.org/terms/gd.html), Follow-The-Leader, [reinforcement learning](https://dictionaryofml.org/terms/reinforcementlearning.html).

## References

- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Murphy (2012). Machine Learning: A Probabilistic Perspective. *MIT Press*.
- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Vapnik (2000). The Nature of Statistical Learning Theory. *Springer*. <https://doi.org/10.1007/978-1-4757-3264-1>
- Goodfellow et al. (2016). Deep Learning. *MIT Press*.
- Bauschke and Combettes (2011). Convex Analysis and Monotone Operator Theory in Hilbert Spaces. *Springer Science+Business Media*. <https://doi.org/10.1007/978-1-4419-9467-7>
- Boyd and Vandenberghe (2004). Convex Optimization. *Cambridge Univ. Press*. <https://doi.org/10.1017/CBO9780511804441>
- Hazan (2022). Introduction to Online Convex Optimization. *MIT Press*. <https://sites.google.com/view/intro-oco/>

## Cite this entry

```bibtex
@incollection{dictml_erm,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {empirical risk minimization (ERM)},
  booktitle = {Dictionary of Applied Machine Learning},
  publisher = {Springer Nature Singapore},
  year = {2026},
  doi = {10.1007/978-981-92-2012-0_4-1},
  isbn = {978-981-92-2012-0},
  url = {https://doi.org/10.1007/978-981-92-2012-0_4-1}
}
```


---

# eigenvalue decomposition (EVD)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-10. [HTML](https://dictionaryofml.org/terms/evd.html) · [typeset PDF](https://dictionaryofml.org/terms/evd.pdf) · [LaTeX](https://dictionaryofml.org/terms/evd.tex) · [Python demo](https://dictionaryofml.org/terms/evd.py) · [notebook](https://dictionaryofml.org/terms/evd.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

An eigenvalue decomposition (EVD) of a square matrix breaks the linear map it represents into elementary operations: expanding a vector in the eigenvectors of the matrix, multiplying each expansion coefficient by the corresponding eigenvalue, and summing the scaled terms up again. A matrix that has an EVD is called diagonalizable; symmetric matrices and matrices with distinct eigenvalues are diagonalizable. A dataset with numeric feature vectors can be characterized by the EVD of its sample covariance matrix. The EVD can be used to speed up computations such as the gradient step of gradient descent methods. For linear regression, each gradient descent iteration then reduces to element-wise operations. The EVD of the Laplacian matrix of an undirected graph is used by spectral clustering.

## Definition

Each day of 2024 at the weather station Krems an der Donau (Austria) is a data point recorded with two features, its lowest and its highest temperature, stacked into a feature vector ${\mathbf x} = \big( x_{1}, x_{2} \big)^{\top} \in \mathbb{R}^{2}$. Fig. 1 shows $20$ of the $m = 366$ days ${\mathbf x}^{(1)}, \ldots, {\mathbf x}^{(m)}$ as a scatterplot together with an ellipse that is computed from their sample mean and sample covariance matrix. The sample mean of the days is

$$
\widehat{\boldsymbol \mu} = \frac{1}{m}
\sum_{r=1}^{m} {\mathbf x}^{(r)} \text{,}
$$

and their sample covariance matrix is

$$
\widehat{{\mathbf C}} = \frac{1}{m}
\sum_{r=1}^{m}
\big( {\mathbf x}^{(r)} - \widehat{\boldsymbol \mu} \big)
\big( {\mathbf x}^{(r)} - \widehat{\boldsymbol \mu} \big)^{\top}
\in \mathbb{R}^{2 \times 2} \text{.}
$$

The shape of the ellipse, which outlines the dataset, is determined by an EVD of the sample covariance matrix $\widehat{{\mathbf C}}$, whose eigenvectors are the axes of the ellipse.

![Figure 1 of the entry evd](https://dictionaryofml.org/terms/fig/evd-fig1.svg)

*Figure 1: Every $19$th of the $366$ days of 2024 at Krems an der Donau, $20$ days in all, drawn as gray dots. Each day is a data point with two features: the lowest and the highest temperature of that day. The ellipse is computed from the sample mean (cross) and the sample covariance matrix of all $366$ days. The solid arrow and the dashed arrow are the two eigenvectors of that sample covariance matrix, scaled to the half-axes of the ellipse, $2\sqrt{\lambda_{j}}\,{\mathbf v}^{(j)}$. Data generated by pythondemos/evd.py*

An EVD for a square matrix ${\mathbf A} \in \mathbb{R}^{d \times d}$ is a factorization of the form

$$
{\mathbf A} = \mathbf{V} {\boldsymbol \Lambda} \mathbf{V}^{-1} \text{.}
$$

The columns of the matrix $\mathbf{V} = \big( {\mathbf v}^{(1)}, \,\ldots, \,{\mathbf v}^{(d)} \big)$ are eigenvectors of the matrix ${\mathbf A}$. The diagonal matrix ${\boldsymbol \Lambda} = {\rm diag} \big\{ \lambda_{1}, \,\ldots, \,\lambda_{d} \big\}$ contains the eigenvalues $\lambda_{j}$ corresponding to the eigenvectors ${\mathbf v}^{(j)}$. Multiplying the factorization by ${\mathbf v}^{(j)}$ gives ${\mathbf A} {\mathbf v}^{(j)} = \lambda_{j} {\mathbf v}^{(j)}$: each eigenvector is a direction that ${\mathbf A}$ only scales, by the factor $\lambda_{j}$; a single such pair satisfies ${\mathbf A} {\mathbf x} = \lambda {\mathbf x}$, and the EVD collects $d$ of them, indexed by $j$.

The EVD breaks the linear map ${\mathbf u} \mapsto {\mathbf A} {\mathbf u}$ into three elementary operations. A vector ${\mathbf u}$ is expanded in the eigenvectors, ${\mathbf u} = \sum_{j=1}^{d}
\beta_{j} {\mathbf v}^{(j)}$ with the expansion coefficients $\boldsymbol{\beta} = \mathbf{V}^{-1} {\mathbf u}$; each coefficient is multiplied by its eigenvalue; and the scaled terms are summed up again, ${\mathbf A} {\mathbf u} = \sum_{j=1}^{d} \lambda_{j}
\beta_{j} {\mathbf v}^{(j)}$.

Unlike the singular value decomposition, which exists for every matrix, square or not (Horn and Johnson, 2013, Thm. 2.6.3), the EVD is defined for square matrices only, and it need not exist. Moreover, the same matrix might allow for different EVDs: any nonzero linear combination of eigenvectors with the same eigenvalue is again an eigenvector with this eigenvalue. The geometric multiplicity of an eigenvalue is the maximum number of linearly independent eigenvectors with this eigenvalue (Horn and Johnson, 2013, Sect. 1.4).

Matrices that allow for an EVD are called diagonalizable. One example of a diagonalizable matrix is

$$
{\mathbf A} = \begin{pmatrix} 1.3 & 0.8 \\ 0.4 & 0.9 \end{pmatrix}
\text{.}
$$

Fig. 2 illustrates its EVD.

![Figure 2 of the entry evd](https://dictionaryofml.org/terms/fig/evd-fig2.svg)

*Figure 2: The matrix ${\mathbf A} \in \mathbb{R}^{2 \times 2}$ from the display above has the eigenvalues $\lambda_{1} = 1.7$ and $\lambda_{2} = 0.5$, with the eigenvectors ${\mathbf v}^{(1)}$ along $(2,1)^{\top}$ and ${\mathbf v}^{(2)}$ along $(-1,1)^{\top}$ (solid arrows). The dashed arrows are the images under ${\mathbf A}$, and each dotted line carries one solid arrow. The eigenvector ${\mathbf v}^{(1)}$ stays on its line, stretched to the longer dashed arrow ${\mathbf A} {\mathbf v}^{(1)} = \lambda_{1} {\mathbf v}^{(1)}$; the eigenvector ${\mathbf v}^{(2)}$ stays on its line, shrunk to the shorter dashed arrow ${\mathbf A} {\mathbf v}^{(2)} = \lambda_{2} {\mathbf v}^{(2)}$. The vector ${\mathbf u} = {\mathbf v}^{(1)} + {\mathbf v}^{(2)}$ leaves its dotted line: its image ${\mathbf A} {\mathbf u} = \lambda_{1} {\mathbf v}^{(1)} + \lambda_{2} {\mathbf v}^{(2)}$ points elsewhere, because its two expansion coefficients are scaled by different factors*

Two sufficient conditions for a real-valued square matrix to be diagonalizable are that it is a symmetric matrix or that its $d$ eigenvalues are distinct (Golub and Loan, 2013; Horn and Johnson, 2013, Thm. 1.3.9, Thm. 4.1.5). For a real-valued symmetric matrix, the EVD has additional structure: the eigenvectors can be chosen to form an orthonormal basis, i.e., the matrix $\mathbf{V}$ in the factorization can be chosen to be orthogonal (Horn and Johnson, 2013, Thm. 4.1.5). The condition that is both necessary and sufficient is that every root of the minimal polynomial of the matrix is simple (Horn and Johnson, 2013, Cor. 3.3.10). For a complex-valued matrix, the first condition is that it is Hermitian; a complex symmetric matrix need not be diagonalizable (Horn and Johnson, 2013, Thm. 4.1.5, Problem 4.4.P10).

The matrix ${\mathbf A}$ of Fig. 2 is not symmetric, since its off-diagonal entries $0.8$ and $0.4$ differ, but its two eigenvalues $1.7$ and $0.5$ are distinct, so it has an EVD. When neither condition holds, an EVD may not exist: the matrix

$$
{\mathbf N} = \begin{pmatrix} 0 & 1 \\ 0 & 0 \end{pmatrix}
$$

is not symmetric and has the eigenvalue $0$ twice, since the eigenvalues of a triangular matrix are its diagonal entries (Horn and Johnson, 2013, Sect. 1.2, exercise on upper triangular matrices after Def. 1.2.9). The eigenvectors of ${\mathbf N}$ are all located on the line $\big\{ (a, 0)^{\top} : a \in \mathbb{R} \big\}$, so there is no invertible matrix $\mathbf{V}$ whose columns are eigenvectors of ${\mathbf N}$.

The ellipse in Fig. 1 is the set of feature vectors ${\mathbf x}$ for which the quadratic form $({\mathbf x} - \widehat{\boldsymbol \mu})^{\top}
\widehat{{\mathbf C}}^{-1} ({\mathbf x} - \widehat{\boldsymbol \mu})$ equals $4$. This set can be characterized more conveniently using an EVD of the sample covariance matrix $\widehat{{\mathbf C}}$. The matrix $\widehat{{\mathbf C}}$ is symmetric, so it has an EVD $\widehat{{\mathbf C}} = \mathbf{V} {\boldsymbol \Lambda} \mathbf{V}^{\top}$ with an orthogonal matrix $\mathbf{V}$ that contains the eigenvectors in its columns. Transforming the feature vector ${\mathbf x}$ of a day according to $\widetilde{{\mathbf x}} := \mathbf{V}^{\top} ({\mathbf x} - \widehat{\boldsymbol \mu})$ turns the quadratic form into

$$
({\mathbf x} - \widehat{\boldsymbol \mu})^{\top}
\widehat{{\mathbf C}}^{-1}
({\mathbf x} - \widehat{\boldsymbol \mu})
= \widetilde{{\mathbf x}}^{\top} {\boldsymbol \Lambda}^{-1}
\widetilde{{\mathbf x}}
= \frac{\widetilde{x}_{1}^{2}}{\lambda_{1}}
+ \frac{\widetilde{x}_{2}^{2}}{\lambda_{2}} \text{,}
$$

since $\widehat{{\mathbf C}}^{-1} = \mathbf{V} {\boldsymbol \Lambda}^{-1} \mathbf{V}^{\top}$. The ellipse is therefore the set where $\widetilde{x}_{1}^{2}/\lambda_{1} + \widetilde{x}_{2}^{2}/\lambda_{2} = 4$, with half-axes $2\sqrt{\lambda_{1}}$ and $2\sqrt{\lambda_{2}}$ along the coordinate axes of $\widetilde{{\mathbf x}}$, that is, along the eigenvectors. The transformed days have the sample covariance matrix $\mathbf{V}^{\top} \widehat{{\mathbf C}} \mathbf{V} = {\boldsymbol \Lambda}$: the two new coordinates have sample covariance zero, and the eigenvalues are their sample variances. Replacing each day by its first coordinate $\widetilde{x}_{1}$ alone keeps the direction of largest variance, which is the idea of principal component analysis (Bishop, 2006, Sect. 12.1.1).

Besides describing the shape of a dataset, the EVD can be used to speed up the implementation of a machine learning method. Consider gradient descent for linear regression, with the update ${\mathbf w}^{(t+1)} = {\mathbf w}^{(t)} -
(2\eta/m) {\mathbf X}^{\top}
\big( {\mathbf X} {\mathbf w}^{(t)} - {\mathbf y} \big)$ for $t = 0,1,\ldots$, with the feature matrix ${\mathbf X}$ and label vector ${\mathbf y}$ of the training set (see linear regression). The matrix ${\mathbf X}^{\top}{\mathbf X}$ is symmetric, so it has an EVD ${\mathbf X}^{\top}{\mathbf X} = \mathbf{V} {\boldsymbol \Lambda} \mathbf{V}^{\top}$ with an orthogonal matrix $\mathbf{V}$. In the transformed coordinates $\widetilde{{\mathbf w}}^{(t)} := \mathbf{V}^{\top} {\mathbf w}^{(t)}$, the update decouples into

$$
\widetilde{w}_{j}^{(t+1)} =
\big( 1 - (2\eta/m) \lambda_{j} \big)
\widetilde{w}_{j}^{(t)}
+ (2\eta/m)
\big( \mathbf{V}^{\top} {\mathbf X}^{\top} {\mathbf y} \big)_{j}
\text{.}
$$

After the one-time computation of the EVD, each iteration is an element-wise multiplication and addition of vectors of length $d$ (pythondemos/evd.py).

Another application of the EVD in machine learning is the clustering of graph-structured data (Luxburg, 2007). Consider data points represented as the nodes $i = 1, \,\ldots, \,n$ of an undirected graph $\mathcal{G}$, e.g., the users of a social network with edges given by friendships. The Laplacian matrix ${\mathbf L}^{(\mathcal{G})}$ of the graph $\mathcal{G}$ is symmetric and positive semi-definite (Luxburg, 2007, Prop. 1). It has an EVD ${\mathbf L}^{(\mathcal{G})} = \mathbf{V} {\boldsymbol \Lambda} \mathbf{V}^{\top}$ with an orthogonal matrix $\mathbf{V}$ and nonnegative eigenvalues.

If the graph $\mathcal{G}$ is connected, the EVD has additional structure: the eigenvalue $0$ has multiplicity one, with the all-ones vector as its eigenvector (Luxburg, 2007, Prop. 2). Moreover, there is then a distinguished eigenvector, called the Fiedler vector: an eigenvector of the smallest nonzero eigenvalue, the algebraic connectivity of $\mathcal{G}$ (Fiedler, 1973). The Fiedler vector can be used for clustering the nodes of the graph, which is the idea of spectral clustering (Luxburg, 2007). Fig. 3 shows the Fiedler vector of a connected graph with six nodes: two clusters of three nodes and a single edge joining the two clusters.

![Figure 3 of the entry evd](https://dictionaryofml.org/terms/fig/evd-fig3.svg)

*Figure 3: (a) A connected undirected graph with six nodes: the cluster of nodes $1$, $2$, $3$ and the cluster of nodes $4$, $5$, $6$, each node connected to the other two of its cluster. The two clusters are joined by the single edge between nodes $3$ and $4$. (b) The Fiedler vector ${\mathbf v}^{(2)}$ of the Laplacian matrix of the graph in (a), as a column vector; a thin dotted line connects each node $i$ to its entry $v^{(2)}_{i}$. The signs of the entries recover the two clusters. Entries computed by pythondemos/evd.py*

**See also:** [matrix](https://dictionaryofml.org/terms/matrix.html), eigenvector, [eigenvalue](https://dictionaryofml.org/terms/eigenvalue.html), diagonalizable, invertible, orthogonal, Laplacian matrix, algebraic connectivity, Fiedler vector, spectral clustering, [gradient descent](https://dictionaryofml.org/terms/gd.html), [linear regression](https://dictionaryofml.org/terms/linreg.html), sample covariance matrix, [principal component analysis](https://dictionaryofml.org/terms/pca.html), spectral decomposition, [singular value decomposition](https://dictionaryofml.org/terms/svd.html), minimal polynomial.

## References

- Horn and Johnson (2013). Matrix Analysis. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781139020411>
- Golub and Loan (2013). Matrix Computations. *The Johns Hopkins Univ. Press*.
- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Luxburg (2007). A tutorial on spectral clustering. *Statist. Comput.*. <https://doi.org/10.1007/s11222-007-9033-z>
- Fiedler (1973). Algebraic connectivity of graphs. *Czechoslovak Mathematical Journal*.

## Cite this entry

```bibtex
@misc{dictml_evd,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {eigenvalue decomposition (EVD)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/evd.html}
}
```


---

# explainability

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/explainability.html) · [typeset PDF](https://dictionaryofml.org/terms/explainability.pdf) · [LaTeX](https://dictionaryofml.org/terms/explainability.tex) · [Python demo](https://dictionaryofml.org/terms/explainability.py) · [notebook](https://dictionaryofml.org/terms/explainability.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Explainability is the property of a machine learning method whose explanations let a human user anticipate the predictions it delivers. The method delivers an explanation along with every prediction, and that explanation is effective if it lets the user anticipate the prediction made for a data point; comprehending the explanation is not required for this test. Explainability is always relative to a specific user or group of users: an explanation can be effective for one and useless for another. Explainability can be measured by comparing the predictions of a learned hypothesis to the anticipations of a user before and after they are provided with an explanation.

## Definition

A radiologist who is shown which pixels of a scan led an image classifier to its prediction can tell, for the next scan, what the classifier will say. A machine learning method is explainable for a human user if the explanations it provides let the user anticipate the predictions it delivers (Colin et al., 2022; Jung and Nardelli, 2020). Explainability thus includes the notion of an explanation: each prediction is delivered with an explanation for this specific prediction (see Fig. 1), such as the feature values that contributed most to it or, for image data, the relevant pixels. In the figure, the explanation for the prediction at a data point with feature vector ${\mathbf x}$ is a linear map that approximates the learned hypothesis near ${\mathbf x}$. The definition asks for the existence of an explanation, not for a particular one: explainability is certified by exhibiting one explanation that works, and one explanation the user cannot follow does not show that the method is unexplainable. Explainability is relative to a specific user or group of users: the same learned hypothesis can be explainable for one user and inscrutable for another.

![Figure 1 of the entry explainability](https://dictionaryofml.org/terms/fig/explainability-fig1.svg)

*Figure 1: Explainability refers to explanations. The learned hypothesis $\widehat{h}$ delivers the prediction $\widehat{h}({\mathbf x})$ for a data point with feature vector ${\mathbf x}$. An explanation is constructed from a simple hypothesis $g$, e.g., a linear map, that approximates $\widehat{h}$ near ${\mathbf x}$. The explanation is delivered to the user along with the prediction*

The usefulness of an explanation can be measured by how much it enables the user to anticipate the predictions on a curated test set: the user states an anticipation for each data point in the test set. If the user comprehends the explanations, these anticipations should be close to the predictions of the learned hypothesis (Colin et al., 2022). A measure of this form is referred to as predictability (International Organization for Standardization and International Electrotechnical Commission, 2022, Sect. 5.15.7). As with interpretability, predictability is weaker than comprehension of the explanations: anticipating the predictions does not require comprehending the provided explanations.

A second measure of this predictability uses the conditional differential entropy, which is defined for RVs. If the data point is modeled as the realization of a random variable, the prediction and the anticipation of the user are RVs too, and their joint probability distribution yields the conditional differential entropy of the predictions given the anticipations (Jung and Nardelli, 2020, Sect. II). The conditional differential entropy quantifies the uncertainty that remains about the predictions once the anticipations are known, so a smaller value means that the anticipations determine the predictions more tightly (Cover and Thomas, 2006, Sect. 8.4). In practice, this conditional differential entropy is unknown and must be replaced by an estimator. One such estimator bins the predictions and the anticipations on a test set, and takes the fraction of test data points that fall into each pair of bins as the joint probability distribution. A related measure, the mutual information between a selected subset of the features and the prediction, is maximized by the explainer of Chen et al. (2018, Sect. 2.1).

An explanation raises explainability when the user can reason with it. Fig. 2 illustrates this for a user who reasons in terms of linear maps and anticipates the predictions of an opaque hypothesis (Gaussian kernel ridge regression, a kernel method). Without explanations, the user anticipates by a single linear map fitted to a few predictions, and the anticipations deviate strongly from the predictions. The explanation for a test data point with feature value $x_{0}$ is a local linear approximation of the hypothesis around it (cf. local interpretable model-agnostic explanations), a slope and an intercept that the user evaluates at $x_{0}$. With this explanation, the anticipations match the predictions almost exactly. The example presupposes that the user knows how to apply a linear approximation: the anticipations improve only because the user can evaluate the explanation for a data point. A user who cannot gains nothing from it.

![Figure 2 of the entry explainability](https://dictionaryofml.org/terms/fig/explainability-fig2.svg)

*Figure 2: An opaque hypothesis $\widehat{h}$ (Gaussian kernel ridge regression, solid curve) and a user who reasons in terms of linear maps. (a) Without explanations the user anticipates by a single linear map fitted to a few predictions (dashed line), which at the test data point $x_{0}$ is far off (open square). The explanation for $x_{0}$ is a local linear approximation of $\widehat{h}$ (thick gray segment); the user evaluates it at $x_{0}$ and anticipates almost exactly (filled circle). (b) The same two anticipations against the predictions for $20$ of the $40$ data points of the test set (every second one). Without explanations (open squares) they deviate strongly; with the local linear approximation as explanation (filled circles) they track the diagonal of perfect anticipation. The estimated conditional differential entropy of the predictions given the anticipations, over all $40$, drops from 1.22 to 0.09 bits. Data generated by pythondemos/explainability.py*

An explanation can also mislead. A computed explanation can be unfaithful to the learned hypothesis, so that anticipations based on it are confidently wrong (Rudin, 2019). Saliency maps, intensity maps over the pixels of an input image (see explainable artificial intelligence), barely change for some methods when the trained deep net is replaced by one with random weights (Adebayo et al., 2018, Sect. 3). The predictability test detects such an explanation: anticipations drawn from it deviate from the predictions.

The international terminology standard for artificial intelligence defines explainability as the property of an artificial intelligence system to express important factors influencing its results in a way that humans can understand. An explanation is intended to answer the question of why the artificial intelligence system produced a result, without arguing that the result was optimal (International Organization for Standardization and International Electrotechnical Commission, 2022, Sect. 3.5.7). The AI Risk Management Framework of the US National Institute of Standards and Technology (NIST) ties explainability to a representation of the mechanisms underlying the operation of an artificial intelligence system. NIST reserves interpretability for the meaning of the output in the context of the designed purpose (National Institute of Standards and Technology, 2023). The two documents thus divide the terms differently. The expressed factors influencing a result, which the terminology standard places under explainability, are close to the NIST notion of interpretability; the NIST notion of explainability is instead close to interpretability as comprehension of the computation of the method.

Regulation treats explainability as a component of transparency. The EU AI Act requires that high-risk AI systems are sufficiently transparent to enable deployers to interpret a system's output and use it appropriately (European Parliament and Council of the European Union, 2024, Art. 13(1)). For individual automated decisions, the right to explanation entitles an affected person to a clear and meaningful account of the role that an artificial intelligence system played in the decision (European Parliament and Council of the European Union, 2024, Art. 86). The Colorado Automated Decision-Making Technology Act of 2026 requires that the deployer give the affected consumer a plain-language description of the decision and of the role the technology played in it, together with the means to request further information (Colorado General Assembly, 2026, Sect. 6-1-1704(3)). What counts as an automated decision is drawn widely: the Court of Justice of the European Union held that a credit information agency computing a person's ability to meet future payments already makes one, when a third party draws strongly on that value (Court of Justice of the European Union, 2023).

**See also:** [explanation](https://dictionaryofml.org/terms/explanation.html), [interpretability](https://dictionaryofml.org/terms/interpretability.html), [explainable artificial intelligence](https://dictionaryofml.org/terms/xaiterm.html), explainable empirical risk minimization, local interpretable model-agnostic explanations, [transparency](https://dictionaryofml.org/terms/transparency.html), right to explanation, [trustworthy artificial intelligence](https://dictionaryofml.org/terms/trustAI.html), [regularization](https://dictionaryofml.org/terms/regularization.html).

## References

- Colin et al. (2022). What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods. *Adv. Neural Inf. Process. Syst.*. <https://proceedings.neurips.cc/paper_files/paper/2022/hash/13113e938f2957891c0c5e8df811dd01-Abstract-Conference.html>
- Jung and Nardelli (2020). An Information-Theoretic Approach to Personalized Explainable Machine Learning. *IEEE Signal Process. Lett.*. <https://doi.org/10.1109/lsp.2020.2993176>
- International Organization for Standardization and International Electrotechnical Commission (2022). ISO/IEC 22989:2022 --- Information technology --- Artificial intelligence --- Artificial intelligence concepts and terminology. <https://www.iso.org/standard/74296.html>
- Cover and Thomas (2006). Elements of Information Theory. *Wiley*. <https://doi.org/10.1002/047174882X>
- Rudin (2019). Stop explaining black box machine learning models for high-stakes decisions and use interpretable models instead. *Nature Mach. Intell.*. <https://doi.org/10.1038/s42256-019-0048-x>
- Adebayo et al. (2018). Sanity Checks for Saliency Maps. *Adv. Neural Inf. Process. Syst. (NeurIPS)*. <https://proceedings.neurips.cc/paper_files/paper/2018/hash/294a8ed24b1ad22ec2e7efea049b8737-Abstract.html>
- National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). <https://doi.org/10.6028/NIST.AI.100-1>
- European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act) (Text with EEA relevance). <https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng>
- Colorado General Assembly (2026). Senate Bill 26-189 --- Automated Decision-Making Technology. <https://leg.colorado.gov/bills/sb26-189>
- Court of Justice of the European Union (2023). Judgment of 7 December 2023, SCHUFA Holding (Scoring), C-634/21, ECLI:EU:C:2023:957. <https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:62021CJ0634>
- Chen et al. (2018). Learning to Explain: An Information-Theoretic Perspective on Model Interpretation. *Proc. 35th Int. Conf. Mach. Learn.*. <https://proceedings.mlr.press/v80/chen18j.html>

## Cite this entry

```bibtex
@misc{dictml_explainability,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {explainability},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/explainability.html}
}
```


---

# explanation

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-06. [HTML](https://dictionaryofml.org/terms/explanation.html) · [typeset PDF](https://dictionaryofml.org/terms/explanation.pdf) · [LaTeX](https://dictionaryofml.org/terms/explanation.tex) · [Python demo](https://dictionaryofml.org/terms/explanation.py) · [notebook](https://dictionaryofml.org/terms/explanation.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

An explanation accompanies a prediction delivered by a machine learning method and states which properties of the data point led to that prediction. Explanations can be text, a score per feature, a simple hypothesis that approximates the learned one near the data point, or a heat map drawn over an image. Two requirements can be contradictory. On the one hand, an explanation must be faithful: it must reflect the computation by which the learned hypothesis arrives at the prediction. On the other hand, it must be understandable: its user must be able to anticipate the predictions it accompanies. An explanation that agreed with the learned hypothesis everywhere would be that hypothesis again, so a simpler one may have to give up faithfulness somewhere.

## Definition

A weather service forecasts frost for the coming night at Krems an der Donau and adds which of the recent measurements weighed most. That addition is an explanation: it is provided alongside the prediction delivered by a machine learning method to increase its transparency for its human user. Explanations can take different forms. For instance, they may consist of human-readable text or quantitative indicators, such as feature importance scores for the individual features of a given data point (Molnar, 2025), obtained, e.g., via the weights of a local linear approximation (Ribeiro et al., 2016).

Fig. 1 illustrates two types of explanations for a learned hypothesis $\widehat{h}(x)$ of a single feature $x$. The first is a local linear approximation $g(x)$ of the nonlinear $\widehat{h}(x)$ around a specific feature value $x'$. The second form of explanation depicted in the figure is a small set of predictions $\widehat{h}(x^{(1)}), \widehat{h}(x^{(2)}), \widehat{h}(x^{(3)})$ at selected feature values, possibly chosen by the user.

![Figure 1 of the entry explanation](https://dictionaryofml.org/terms/fig/explanation-fig1.svg)

*Figure 1: A learned hypothesis $\widehat{h}(x)$ of a single feature $x$, explained locally at the point $x'$ by a linear approximation $g(x)$ (thick gray segment), and by its values $\widehat{h}\big(x^{(r)} \big)$ for $r=1, \,2, \,3$ (filled dots); dashed lines drop from each point to its feature value*

Another form of explanation is a heat map: an intensity map that scores each region of an image by how much it contributed to the prediction, drawn over the image itself (Selvaraju et al., 2017). For a convolutional neural network, those scores are computed from the network's own activations, which is what a class activation map does.

Explanations differ in scope. The local linear approximation of local interpretable model-agnostic explanations concerns a single prediction; a global explanation, such as a decision tree fitted to the predictions of the learned hypothesis, describes its behavior across the whole feature space (Molnar, 2025). A counterfactual answers a different question: not which features drove the prediction, but what change of the feature vector would have changed it (Wachter et al., 2018).

Whatever form it takes, an explanation carries two requirements: it must be (i) understandable and (ii) faithful. Understandable means that the user can work with the explanation on their own. Faithful means that the explanation reflects the computation the learned hypothesis carries out, rather than one that is easier to present. An intensity map over the pixels of an image is faithful when manipulating a few of the pixels it highlights changes the prediction while changing dark ones does not (see explainable artificial intelligence).

These two requirements can be contradictory. An unfaithful explanation can even be understandable and look plausible while carrying nothing about the computation. For some saliency methods, the map computed from a network with random weights is nearly the same as the map computed from the trained network. Such a map traces the edges of the input image and carries nothing about the prediction, yet it looks like a plausible explanation (Adebayo et al., 2018). An explanation that agreed with the learned hypothesis everywhere would be that hypothesis again, so a simpler explanation may have to give up faithfulness somewhere.

Fig. 2 shows both requirements at work on a prediction made from weather radar. The features are the hourly precipitation over a 120 km box around Krems an der Donau, and the prediction answers whether it will rain at Krems two hours later. A convolutional neural network trained on these images is explained by a class activation map (Selvaraju et al., 2017). For the hour drawn, the network assigns rain a probability close to one, and the class activation map scores the bands of precipitation highest south and south-east of the town rather than on the rain already overhead. Setting the precipitation to zero in the 360 cells with the highest class activation map scores lowers the predicted score by 1.57, against 0.26 for the 360 cells with the lowest scores, and the same holds on 63 of the 72 held-out hours.

The same content can be delivered as text. The sentence under the panels is generated from the class activation map by computing the radius that holds the cells it scores highest and the direction of their center of mass. It is therefore faithful by construction and can be checked against the radar image on the left.

![Figure 2 of the entry explanation](https://dictionaryofml.org/terms/fig/explanation-fig2.svg)

*Figure 2: An explanation for a prediction made from weather radar on 15 September 2024 at 08:00 UTC. (a) The observed precipitation over a 120 km box around Krems an der Donau (circle), from which a convolutional neural network predicts rain at Krems two hours later. (b) The class activation map, which scores each cell by how much it contributed to the prediction; the highest scores lie on the bands south and south-east of the town rather than on the rain overhead. Below: a textual explanation generated from the class activation map, naming the radius and direction of the cells it scores highest. Data generated by pythondemos/explanation.py*

**See also:** machine learning, prediction, [feature](https://dictionaryofml.org/terms/feature.html), [data point](https://dictionaryofml.org/terms/datapoint.html), [classification](https://dictionaryofml.org/terms/classification.html), [explainability](https://dictionaryofml.org/terms/explainability.html), [explainable artificial intelligence](https://dictionaryofml.org/terms/xaiterm.html), [interpretable machine learning](https://dictionaryofml.org/terms/interpretableml.html), local interpretable model-agnostic explanations, SHapley Additive exPlanations, counterfactual, class activation map.

## References

- Molnar (2025). Interpretable Machine Learning: A Guide for Making Black Box Models Explainable. *Ebook*. <https://christophm.github.io/interpretable-ml-book/>
- Ribeiro et al. (2016). ``Why Should I Trust You?'': Explaining the Predictions of Any Classifier. *Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining*. <https://doi.org/10.1145/2939672.2939778>
- Selvaraju et al. (2017). Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. *2017 IEEE Int. Conf. Comput. Vis.*. <https://doi.org/10.1109/ICCV.2017.74>
- Wachter et al. (2018). Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR. *Harvard Journal of Law & Technology*. <https://doi.org/10.2139/ssrn.3063289>
- Adebayo et al. (2018). Sanity Checks for Saliency Maps. *Adv. Neural Inf. Process. Syst. (NeurIPS)*. <https://proceedings.neurips.cc/paper_files/paper/2018/hash/294a8ed24b1ad22ec2e7efea049b8737-Abstract.html>

## Cite this entry

```bibtex
@misc{dictml_explanation,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {explanation},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/explanation.html}
}
```


---

# fairness (machine learning)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/fairness.html) · [typeset PDF](https://dictionaryofml.org/terms/fairness.pdf) · [LaTeX](https://dictionaryofml.org/terms/fairness.tex) · [Python demo](https://dictionaryofml.org/terms/fairness.py) · [notebook](https://dictionaryofml.org/terms/fairness.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Fairness (machine learning) is the property of a learned hypothesis, or of the decision rule built on it, to treat data points with different values of a sensitive attribute, such as gender or age, in a comparable way. A counterfactual criterion requires the prediction to be unchanged if the sensitive attribute had taken another value. An individual criterion requires similar predictions for data points that are similar under a task-specific metric. Group criteria equate a measure across the groups: the approval rates (demographic parity), the true-positive and false-positive rates (equalized odds), or the probability of the label given the score (calibration). The criteria are incompatible in general: calibration and equalized odds cannot both hold when the groups differ in their share of positive labels. A chosen criterion is met by a design choice for a component of empirical risk minimization: excluding the sensitive attribute and the features correlated with it from the model, adding a penalty term or a constraint to the loss function, or resampling the training set.

## Definition

Consider a bank that approves a loan when the creditworthiness predicted by a learned hypothesis $\widehat{h}$ exceeds a threshold $\tau$. The feature vector ${\mathbf x}$ of an applicant collects the income, the gender, and further attributes, and $\widehat{h}$ was learned from the training set $\mathcal{D}^{(\rm train)}$ of five completed loans. An applicant with feature vector ${\mathbf x}'$ obtains $\widehat{h}({\mathbf x}') = 2.94$, below $\tau = 3.8$, and is rejected. The same applicant with only the gender flipped, ${\mathbf x}^{\star}$, obtains $\widehat{h}({\mathbf x}^{\star})
= 3.93$ and is approved. Whether this system is fair has several precise answers, and they cannot all hold at once. Fairness in machine learning is the property of a learned hypothesis, or of the decision rule built on it, to treat data points with different values of a sensitive attribute $s$, such as gender, race, or age, in a comparable way, in addition to making accurate predictions (Barocas et al., 2023). It is one of the key requirements for trustworthy artificial intelligence.

Three families of fairness criteria are distinguished (Barocas et al., 2023). A *counterfactual* criterion requires that the prediction for a data point would not change if its sensitive attribute had taken another value, under a causal model of how $s$ affects the other features (see causality) (Kusner et al., 2017). An *individual* criterion requires that data points that are similar under a task-specific metric, such as one that ignores $s$, receive similar predictions (Dwork et al., 2012). A *group* criterion requires equality of some measure across the groups defined by $s$. Demographic parity equates the approval rates,

\begin{equation}
\tag{1}
\mathbb{P}\left(\widehat{h}({\mathbf x}) \geq \tau \mid s = 0\right)
= \mathbb{P}\left(\widehat{h}({\mathbf x}) \geq \tau \mid s = 1\right)
\text{,}
\end{equation}

where the data point is modeled as the realization of a random variable. For a binary label $y \in \{0,1\}$, such as whether a loan was repaid, equalized odds equates the true-positive rate and the false-positive rate across the groups (Hardt et al., 2016), and calibration equates $\mathbb{P}\left(y = 1 \mid \widehat{h}({\mathbf x}) = c, s\right)$ across the groups for every score $c$ (Chouldechova, 2017).

Fig. 1 shows the credit-scoring example. The horizontal axis is a one-dimensional sketch of the feature vector; the gender flip moves the applicant from ${\mathbf x}'$ (open square) to ${\mathbf x}^{\star}$ (open triangle). The gap $\widehat{h}({\mathbf x}^{\star}) -
\widehat{h}({\mathbf x}') = 0.99$ (double arrow) is caused by the gender alone, so $\widehat{h}$ fails the counterfactual criterion. It fails the individual criterion under a metric that ignores $s$ as well: ${\mathbf x}'$ and ${\mathbf x}^{\star}$ are then the same data point, yet they receive predictions on opposite sides of $\tau$. The group criteria cannot be checked from five loans. They compare the predictions of $\widehat{h}$ over the population of applicants, which the probability distribution of the demographic-parity condition Eq. (1) describes.

![Figure 1 of the entry fairness](https://dictionaryofml.org/terms/fig/fairness-fig1.svg)

*Figure 1: The counterfactual fairness criterion in the credit-scoring example. The horizontal axis is a one-dimensional sketch of the feature vector. The solid curve is the hypothesis $\widehat{h}$ learned from the training set $\mathcal{D}^{(\rm train)}$ of five completed loans (filled circles); a loan is approved when the predicted creditworthiness exceeds the threshold $\tau$ (dotted line). The applicant with feature vector ${\mathbf x}'$ (open square) is rejected, while the same applicant with the gender flipped, ${\mathbf x}^{\star}$ (open triangle), is approved. The vertical double arrow spans the gap between the two predictions, which the gender alone causes*

The criteria are mutually incompatible in general. Kleinberg et al. (2017, Thm. 1.1) and Chouldechova (2017, Eq. (2.6)) show that calibration and equalized odds cannot both hold when the fraction of data points with $y = 1$ differs between the groups and $\widehat{h}$ is not perfect. Fairness in machine learning is therefore not a single property to be optimized but a choice among criteria. The choice depends on the application: on the harm caused by a wrong prediction, and on the applicable law. Two examples are Art. 22 of the general data protection regulation on automated decisions (European Parliament and Council of the European Union, 2016, Art. 22) and the duty under the EU AI Act to examine the training set of a high-risk artificial intelligence system for biases that lead to discrimination (European Parliament and Council of the European Union, 2024, Art. 10) (see data governance).

Once a criterion is chosen, it is met by a design choice for one of the components of empirical risk minimization (Pfau and Jung, 2024, Sect. VIII). The model can exclude the sensitive attribute; ${\mathbf x}'$ and ${\mathbf x}^{\star}$ then coincide and the gap in Fig. 1 vanishes. Excluding $s$ alone does not suffice, since features correlated with it, such as the postal code, carry the same information. The loss function can carry a penalty term for the gap, or the gap can be a constraint of the constrained optimization problem that replaces empirical risk minimization. The training set can be resampled so that the groups are represented equally (see imbalanced data). Each choice lowers the gap of the chosen criterion, at a cost. The hypothesis learned with a penalty term or a constraint has an average loss on $\mathcal{D}^{(\rm train)}$ at least that of the empirical risk minimization solution, since empirical risk minimization minimizes this loss over all of $\mathcal{H}$. Lowering one gap can also raise the gap of another criterion: a score that is calibrated within each group and gives the positive class, and likewise the negative class, the same average score in both groups exists only when the groups share the fraction of data points with $y =
1$ or the score predicts perfectly (Kleinberg et al., 2017, Thm. 1.1).

**See also:** sensitive attribute, [trustworthy artificial intelligence](https://dictionaryofml.org/terms/trustAI.html), [hypothesis](https://dictionaryofml.org/terms/hypothesis.html), decision rule, prediction, calibration, metric, causality, counterfactual, [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), penalty term, constrained optimization problem, data governance, EU AI Act, general data protection regulation, high-risk artificial intelligence system, imbalanced data, [explainable artificial intelligence](https://dictionaryofml.org/terms/xaiterm.html).

## References

- Barocas et al. (2023). Fairness and Machine Learning: Limitations and Opportunities. *MIT Press*. <https://fairmlbook.org/>
- Kusner et al. (2017). Counterfactual Fairness. *Advances in Neural Information Processing Systems 30 (NIPS 2017)*. <https://proceedings.neurips.cc/paper/2017/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html>
- Dwork et al. (2012). Fairness through awareness. *Proceedings of the 3rd innovations in theoretical computer science conference*. <https://doi.org/10.1145/2090236.2090255>
- Hardt et al. (2016). Equality of opportunity in supervised learning. *Adv. Neur. Inf. Proc. Syst.*. <https://arxiv.org/abs/1610.02413>
- Chouldechova (2017). Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. *Big data*. <https://doi.org/10.1089/big.2016.0047>
- European Parliament and Council of the European Union (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (Text with EEA relevance). <https://eur-lex.europa.eu/eli/reg/2016/679/oj>
- European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act) (Text with EEA relevance). <https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng>
- Pfau and Jung (2024). Engineering Trustworthy AI: A Developer Guide for Empirical Risk Minimization. *arXiv preprint arXiv:2410.19361*. <https://arxiv.org/abs/2410.19361>
- Kleinberg et al. (2017). Inherent trade-offs in the fair determination of risk scores. *Proceedings of the 8th Innovations in Theoretical Computer Science Conference*. <https://doi.org/10.1145/3033274.3033302>

## Cite this entry

```bibtex
@misc{dictml_fairness,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {fairness (machine learning)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/fairness.html}
}
```


---

# feature learning

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-07. [HTML](https://dictionaryofml.org/terms/featlearn.html) · [typeset PDF](https://dictionaryofml.org/terms/featlearn.pdf) · [LaTeX](https://dictionaryofml.org/terms/featlearn.tex) · [Python demo](https://dictionaryofml.org/terms/featlearn.py) · [notebook](https://dictionaryofml.org/terms/featlearn.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Feature learning is the task of learning, from a dataset, a feature transformation that turns the raw features of a data point into new features that make a subsequent learning task easier. A feature learning method is specified by the two feature spaces, by a hypothesis space of candidate feature transformations, and by a quantitative measure of how useful a feature transformation is, computed from a dataset that need not carry labels. Principal component analysis learns a linear feature transformation to fewer features, judged by the linear reconstruction error, and an autoencoder obtains a nonlinear one by the same criterion. The hidden layers of a deep net form a chain of feature transformations learned jointly with the output layer, so that each layer builds its features from those of the previous one. Pretraining an encoder on unlabeled data points learns features that are reused across learning tasks. In weather forecasting, forty measurements from the five days before are turned into two learned features from which linear regression predicts the maximum temperature of the next day.

## Definition

Consider the task of forecasting the weather of tomorrow at a weather station from the measurements of the previous five days. Each day is a data point, and its label is the maximum temperature of that day. Its raw features are eight daily measurements: the lowest, highest and mean air temperature, the precipitation, the sunshine duration, the humidity, the air pressure and the wind speed. Taken over each of the five previous days, that is a list of forty numbers. A point with forty coordinates cannot be drawn in a scatterplot. A feature transformation that delivers two new features from the forty raw ones places every day in a scatterplot (see Fig. 1), in which the days whose label lies above the median of all labels separate from the others, and a linear model of the two new features may already predict the label. Feature learning is the task of learning such a feature transformation from a dataset instead of designing it by hand.

![Figure 1 of the entry featlearn](https://dictionaryofml.org/terms/fig/featlearn-fig1.svg)

*Figure 1: Twenty of the 361 days of 2024 at Krems an der Donau, every 17th warm day (filled circles) and every 17th cold day (open squares) in calendar order, each placed by two learned features $z_{1}$ and $z_{2}$ in place of its forty raw ones, the eight daily measurements of each of the five previous days. Warm and cold days, split at the median of the label, separate along $z_{1}$, as they do for all 361 days. Data generated by pythondemos/featlearn.py*

Feature learning is the task of learning a feature transformation

$$
{\mathbf \Phi}: \mathcal{X} \rightarrow \mathcal{X}', \qquad
{\mathbf x} \mapsto {\mathbf z} := {\mathbf \Phi}({\mathbf x}) \text{.}
$$

The transformation maps the raw feature vector ${\mathbf x} \in
\mathcal{X}$ of a data point to a new feature vector ${\mathbf z}$ from a new feature space $\mathcal{X}'$. A feature learning method is specified by three design choices: the two feature spaces $\mathcal{X}$ and $\mathcal{X}'$, a hypothesis space $\mathcal{H}$ of candidate feature transformations, and a quantitative measure of how useful a specific ${\mathbf \Phi} \in \mathcal{H}$ is. The measure is computed from a dataset, which need not carry labels, and it need not be the dataset of the learning task at hand. A useful feature transformation is one that makes a subsequent learning task easier (Goodfellow et al., 2016, Ch. 15): the data points become separable by a linear model, fewer features suffice, or the features learned once serve several learning tasks. In the weather example, the learned features are judged afterwards by the validation error of linear regression on them.

A measure of usefulness can be built from a second map. The feature transformation is then read as an encoder, and a decoder ${\mathbf \Phi}^{*}: \mathcal{X}' \rightarrow \mathcal{X}$ rebuilds the raw feature vector from the new one (Goodfellow et al., 2016, Ch. 14). What the encoder kept is then measured by how well the decoder rebuilds the data point from it, the reconstruction error

$$
\frac{1}{m} \sum_{r=1}^{m}
L\Big( {\mathbf x}^{(r)},\,
{\mathbf \Phi}^{*}\big( {\mathbf \Phi}\big({\mathbf x}^{(r)}\big) \big) \Big)
\text{.}
$$

The loss function $L$ penalizes a rebuilt feature vector for differing from the original. A common choice is the squared error loss (Goodfellow et al., 2016, Sect. 14.1). The dataset it is computed on needs no labels: what the decoder has to reproduce is the data point itself.

The decoder may be learned alongside the encoder. The measure is then the minimum of the reconstruction error over a second hypothesis space of decoders. It may instead be fixed, and the encoder alone must then supply what the decoder cannot. Restricting the decoder therefore decides what the new features have to carry.

Principal component analysis is this construction with $\mathcal{X} :=
\mathbb{R}^{d}$, $\mathcal{X}' := \mathbb{R}^{d'}$ for some $d' < d$, and the hypothesis space of linear maps,

$$
\mathcal{H} := \big\{ {\mathbf \Phi}: \mathbb{R}^{d}
\rightarrow \mathbb{R}^{d'}: {\mathbf x} \mapsto {\mathbf W} {\mathbf x}
\text{ with some } {\mathbf W} \in \mathbb{R}^{d' \times d}
\big\} \text{.}
$$

The decoder is linear too and is learned alongside the encoder ${\mathbf W}$. The usefulness of a specific ${\mathbf \Phi}({\mathbf x}) = {\mathbf W} {\mathbf x}$ is therefore the minimum of the reconstruction error over the reconstruction matrices ${\mathbf R}$. On the dataset $\mathcal{D} = \big\{ {\mathbf x}^{(1)},
\ldots, {\mathbf x}^{(m)} \big\}$, that minimum is

$$
\min_{{\mathbf R} \in \mathbb{R}^{d \times d'}}
\sum_{r=1}^{m} \left\lVert {\mathbf x}^{(r)}
- {\mathbf R} {\mathbf W} {\mathbf x}^{(r)} \right\rVert_{2}^{2} \text{.}
$$

For feature vectors with zero sample mean, a best ${\mathbf W}$ has as rows the $d'$ eigenvectors of the sample covariance matrix with the largest eigenvalues (Hastie et al., 2009, Sect. 14.5.1). Only the row space of ${\mathbf W}$ matters: replacing ${\mathbf W}$ by $\mathbf{T} {\mathbf W}$ with an invertible $\mathbf{T} \in \mathbb{R}^{d' \times
d'}$ and ${\mathbf R}$ by ${\mathbf R} \mathbf{T}^{-1}$ leaves the product ${\mathbf R} {\mathbf W}$, and so the reconstruction error, unchanged. An autoencoder keeps the same criterion and replaces both linear maps by ANNs, so that the new features come from a nonlinear feature transformation. With a linear decoder and the squared error loss, it spans the subspace principal component analysis finds (Goodfellow et al., 2016, Sect. 14.1).

Applied to the weather example, principal component analysis delivers the two learned features of Fig. 1. The forty raw features of the 361 days of 2024 at Krems an der Donau are turned into two learned features. Each raw feature is first scaled to zero sample mean and unit sample variance, because temperatures, precipitation and pressure carry different units. Warm and cold days separate along the first learned feature: 89 percent of the warm days have a positive $z_{1}$ and 92 percent of the cold days a nonpositive one. Linear regression on the two learned features is trained on the days from January to August. It then predicts the maximum temperature of the days from September to December with a validation error of 13.5 squared degrees Celsius. Linear regression on all forty raw features reaches 10.2, and predicting the sample mean of the training labels reaches 110.9. The two learned features therefore retain most of the predictive content of the forty raw ones.

A deep net learns a chain of feature transformations. Each hidden layer $\ell = 1, \ldots, L$ is a feature transformation ${\mathbf \Phi}^{(\ell)}$ that reads in the activations ${\mathbf z}^{(\ell-1)}$ of the previous layer, with ${\mathbf z}^{(0)} := {\mathbf x}$. The features delivered to the output layer are the composition ${\mathbf z}^{(L)} = {\mathbf \Phi}^{(L)} \circ \cdots \circ
{\mathbf \Phi}^{(1)} ({\mathbf x})$ (see Fig. 2). The feature transformations are learned jointly with the output layer by empirical risk minimization with the loss of the final prediction, so the usefulness of the learned features is measured by how well the output layer, typically a linear model, predicts the label from ${\mathbf z}^{(L)}$ (Goodfellow et al., 2016, Ch. 15). The chain is useful because each layer builds its features from those of the previous one, so the feature transformations ${\mathbf \Phi}^{(\ell)} \circ \cdots \circ {\mathbf \Phi}^{(1)}$ grow deeper with $\ell$. In image classification, early layers deliver edges, later ones corners and object parts (Goodfellow et al., 2016, Fig. 1.2), and the features at the last hidden layer separate the classes by a linear surface (Bishop and Bishop, 2024, Sect. 6.3.3).

![Figure 2 of the entry featlearn](https://dictionaryofml.org/terms/fig/featlearn-fig2.svg)

*Figure 2: A deep net as a chain of feature transformations: hidden layer $\ell$ turns the activations ${\mathbf z}^{(\ell-1)}$ of the previous layer into ${\mathbf z}^{(\ell)}$, with ${\mathbf z}^{(0)}$ the raw feature vector, and the output layer applies a linear model to ${\mathbf z}^{(L)}$. All layers are learned jointly with the linear model by empirical risk minimization with the loss of the final prediction*

Pretraining an encoder on a large dataset of unlabeled data points, with a loss constructed from the data points themselves (see self-supervised learning), learns features that are reused across learning tasks (Goodfellow et al., 2016, Sect. 15.1). A new learning task appends a task-specific hypothesis with few model parameters to the learned ${\mathbf \Phi}$ and adapts it by fine-tuning on its own dataset. The dataset used to learn ${\mathbf \Phi}$ can then be much larger than any labeled dataset, because unlabeled data points are cheap to collect (Bishop and Bishop, 2024, Sect. 6.3.3).

Alternative constructions of a feature transformation do not learn ${\mathbf \Phi}$ (Goodfellow et al., 2016, Ch. 6). A kernel method fixes ${\mathbf \Phi}$ through the choice of a kernel, with $\mathcal{X}'$ the reproducing kernel Hilbert space $\mathcal{H}_{k}$ and ${\mathbf z} = k\big({\mathbf x},\cdot\big)$, the kernel with its first argument fixed, never evaluated explicitly, and learns only a linear hypothesis on $\mathcal{X}'$. A hand-designed ${\mathbf \Phi}$ encodes domain knowledge instead of a dataset.

**See also:** feature transformation, encoder, decoder, [feature](https://dictionaryofml.org/terms/feature.html), feature space, [hypothesis space](https://dictionaryofml.org/terms/hypospace.html), [principal component analysis](https://dictionaryofml.org/terms/pca.html), [autoencoder](https://dictionaryofml.org/terms/autoencoder.html), [dimensionality reduction](https://dictionaryofml.org/terms/dimred.html), deep net, layer, [pretraining](https://dictionaryofml.org/terms/pretraining.html).

## References

- Goodfellow et al. (2016). Deep Learning. *MIT Press*.
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Bishop and Bishop (2024). Deep Learning: Foundations and Concepts. *Springer Nature*. <https://doi.org/10.1007/978-3-031-45468-4>

## Cite this entry

```bibtex
@misc{dictml_featlearn,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {feature learning},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/featlearn.html}
}
```


---

# feature

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/feature.html) · [typeset PDF](https://dictionaryofml.org/terms/feature.pdf) · [LaTeX](https://dictionaryofml.org/terms/feature.tex) · [Python demo](https://dictionaryofml.org/terms/feature.py) · [notebook](https://dictionaryofml.org/terms/feature.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A feature of a data point is one of its attributes that can be measured or computed easily, without human supervision. The features of a data point are collected into a feature vector, which a hypothesis map reads to predict the label of the data point. The red-green-blue (RGB) pixel intensities of a digital image or the signal values of an audio recording are typical features. Which attributes of a data point serve as features, rather than as labels, is a design choice within a machine learning application.

## Definition

A digital photo of a cow is a grid of pixels, and the red, green and blue intensity of each pixel is a number that a computer obtains without any human looking at the photo. Such a number is a feature of the data point: an attribute of the data point ${\mathbf z}$ that is measured, or computed, without human supervision (Anon., 2003; Everitt and Skrondal, 2010; Gujarati and Porter, 2009). The features of a data point are assembled into a feature vector ${\mathbf x} = {\mathbf \Phi}({\mathbf z})$ by a feature transformation, a function ${\mathbf \Phi}: \mathcal{Z} \rightarrow \mathcal{X}$, viewing the data point itself as its raw features from which the computed features are obtained. A hypothesis map takes the feature vector as input and outputs a prediction of the label of the data point.

Besides the red-green-blue (RGB) intensities of the pixels of a photo, Fig. 1 shows another example of features: the $d$ signal values of a finite-duration audio signal, used as its features $x_{1}, \,\ldots, \,x_{d}$.

![Figure 1 of the entry feature](https://dictionaryofml.org/terms/fig/feature-fig1.svg)

*Figure 1: Audio signal (curve) against time and its signal values at the sampling instants (dots), which can be used as features $x_{1},\,\ldots,\,x_{d}$*

Whether a given attribute is treated as a feature or a label is not inherent to the data point. It is a design choice that depends on the machine learning application and the resources available for determining each attribute. An attribute should be used as a feature only if its value can be determined for every data point at the time the prediction is needed; an attribute that becomes known only afterwards is a candidate for the label instead.

Once the features are chosen, new features can be constructed by transforming existing ones. Such a transformation can be an arithmetic computation applied to the existing features. For example, if a data point has a numeric feature $x \in \mathbb{R}$, the quantities $x^{2}$, $\exp(-x)$, and $3x$ can also be used as features of the data point.

Instead of its raw signal values, the audio signal in Fig. 1 can be described by the magnitudes of its discrete-time Fourier transform (DTFT) (Oppenheim et al., 1999). These features describe the frequency content of the signal and are unchanged by time shifts, since shifting the signal by $n_{0}$ samples multiplies its DTFT by $\exp(-\mathrm{j} \omega n_{0})$, a factor of modulus one. This invariance is useful for a stationary signal, one whose statistical properties do not change over time. Another example is the activation of a neuron within an artificial neural network: it is a new feature derived from the input features by a sequence of basic computations performed by the artificial neural network.

Using activations as features underlies a second, narrower use of the word *feature* in mechanistic interpretability: a feature is a human-interpretable concept encoded inside a trained artificial neural network, such as a curve detector or the presence of a wheel. The activations of a subset of $d$ neurons (for example, a whole layer, or the neurons that an analyst selects for study) form a vector in a Euclidean space $\mathbb{R}^{d}$. Whereas the broader sense above treats a single neuron activation as one feature, here a feature is obtained as a function of this whole activation vector ${\mathbf a}$. One such function is the projection of the activation vector ${\mathbf a}$ onto a one-dimensional subspace spanned by a unit vector (or direction) ${\mathbf u}$ (Olah et al., 2020; Park et al., 2024). This unit vector can be obtained as the unit normal ${\mathbf u}$ of a hyperplane separating data points that express the concept from those that do not (Kim et al., 2018) (see Fig. 2).

![Figure 2 of the entry feature](https://dictionaryofml.org/terms/fig/feature-fig2.svg)

*Figure 2: A feature as a function of an activation vector ${\mathbf a}$ obtained from a trained artificial neural network. (a) An artificial neural network with one hidden layer of three neurons; the activations $a_{1}, a_{2}$ of a selected subset (dashed box) form the activation vector ${\mathbf a}=\big(a_{1},a_{2}\big)^{\top}$, while $a_{3}$ is not used. (b) Each data point maps to a point $(a_{1}, a_{2})^{\top}$ in the activation space. Data points that express the concept (filled circles) and those that do not (open squares) are separated by a hyperplane (dashed line) with unit normal ${\mathbf u}$ (thick arrow). The feature value is the projection of the activation vector onto the line spanned by ${\mathbf u}$ (thin solid line), shown for two points by dotted lines ending in small dots*

Another construction of an activation-based feature is based on regions instead of directions in the activation space (Crabbé and van der Schaar, 2022). In either case, constructing the function that maps the activation vector to the feature requires a set of data points labeled as representing the concept or not.

Beyond their construction, regulation constrains which features may be used and how accurate they must be. The EU AI Act requires the training data of high-risk ML systems to be relevant, sufficiently representative, and, to the best extent possible, free of errors and complete (European Parliament and Council of the European Union, 2024, Art. 10(3)). When features are personal data, the general data protection regulation adds the data minimization principle (European Parliament and Council of the European Union, 2016, Art. 5(1)(c)) and the accuracy principle (European Parliament and Council of the European Union, 2016, Art. 5(1)(d)).

Conversely, deliberately perturbing the features can be useful: adding small perturbations to the features of data points during training is a form of data augmentation that acts as regularization. For example, the ridge regression penalty equals the average loss under Gaussian feature perturbations: for a linear model with model parameters ${\mathbf w}$, adding zero-mean Gaussian noise of variance $\alpha$ to each feature raises the expected squared error loss by $\alpha \left\lVert {\mathbf w} \right\rVert_{2}^{2}$, the ridge penalty (Bishop, 1995).

**Synonyms:** covariate, explanatory variable, independent variable, input, predictor, regressor.

**See also:** [data point](https://dictionaryofml.org/terms/datapoint.html), [label](https://dictionaryofml.org/terms/label.html), [feature vector](https://dictionaryofml.org/terms/featurevec.html), feature space, mechanistic interpretability, [concept activation vector](https://dictionaryofml.org/terms/cav.html), [data augmentation](https://dictionaryofml.org/terms/dataaug.html).

## References

- Anon. (2003). The Oxford Dictionary of Statistical Terms. *Oxford Univ. Press*. <https://doi.org/10.1093/oso/9780198509943.001.0001>
- Everitt and Skrondal (2010). The Cambridge Dictionary of Statistics. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9780511779633>
- Gujarati and Porter (2009). Basic Econometrics. *McGraw-Hill/Irwin*.
- Oppenheim et al. (1999). Discrete-Time Signal Processing. *Prentice Hall*.
- Olah et al. (2020). Zoom In: An Introduction to Circuits. *Distill*. <https://doi.org/10.23915/distill.00024.001>
- Park et al. (2024). The Linear Representation Hypothesis and the Geometry of Large Language Models. *Proc. 41st Int. Conf. Mach. Learn.*. <https://proceedings.mlr.press/v235/park24c.html>
- Kim et al. (2018). Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). *Proc. 35th Int. Conf. Mach. Learn.*. <https://proceedings.mlr.press/v80/kim18d.html>
- Crabbé and van der Schaar (2022). Concept Activation Regions: A Generalized Framework for Concept-Based Explanations. *Adv. Neural Inf. Process. Syst.*. <https://proceedings.neurips.cc/paper_files/paper/2022/hash/11a7f429d75f9f8c6e9c630aeb6524b5-Abstract.html>
- European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act) (Text with EEA relevance). <https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng>
- European Parliament and Council of the European Union (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (Text with EEA relevance). <https://eur-lex.europa.eu/eli/reg/2016/679/oj>
- Bishop (1995). Training with Noise is Equivalent to Tikhonov Regularization. *Neural Computation*. <https://doi.org/10.1162/neco.1995.7.1.108>

## Cite this entry

```bibtex
@incollection{dictml_feature,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {feature},
  booktitle = {Dictionary of Applied Machine Learning},
  publisher = {Springer Nature Singapore},
  year = {2026},
  doi = {10.1007/978-981-92-2012-0_11-1},
  isbn = {978-981-92-2012-0},
  url = {https://doi.org/10.1007/978-981-92-2012-0_11-1}
}
```


---

# feature importance

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-07. [HTML](https://dictionaryofml.org/terms/featureimportance.html) · [typeset PDF](https://dictionaryofml.org/terms/featureimportance.pdf) · [LaTeX](https://dictionaryofml.org/terms/featureimportance.tex) · [Python demo](https://dictionaryofml.org/terms/featureimportance.py) · [notebook](https://dictionaryofml.org/terms/featureimportance.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Feature importance assigns a number to each feature, an attribute of a data point, measuring how strongly a learned hypothesis relies on that feature when it maps a feature vector to a prediction. Coefficients of a linear model describe the prediction change per unit of a feature; their magnitudes depend on the units of the features, which standardizing the features removes. The permutation importance of a feature is the increase of the average loss when its values are permuted across the data points. The score uses the learned hypothesis only through its predictions on a scoring dataset, so it applies to a random forest as well as to a linear model, whereas coefficient magnitudes need a linear hypothesis or a linear approximation of a nonlinear one. The score characterizes one fixed learned hypothesis and additionally depends on a scoring dataset distinct from the training set used to learn that hypothesis. It therefore measures reliance on a feature, not the usefulness of the feature itself: a feature that predicts the label well obtains a score near zero whenever the hypothesis ignores it.

## Definition

The weather station Krems an der Donau (Austria) records six observations of each day: the maximum and the minimum air temperature, the mean air pressure, the mean relative humidity, the sunshine duration, and the precipitation. Each data point is a day, the day whose maximum temperature is to be predicted. Its label $y$ is that maximum temperature, and its twelve features $x_{1},\ldots,x_{12}$ are the six observations of the day before and the six observations of two days before.

Fig. 1 depicts observed maximum daily temperatures along with their predictions obtained from a hypothesis $\widehat{h}$ learned via linear regression. The prediction (or forecast) $\widehat{h}({\mathbf x})$ for a day is computed from the feature vector ${\mathbf x} = \big(x_{1},\ldots,x_{12} \big)^{\top}$. An importance score quantifies how strongly this prediction relies on a particular feature of the data point. It assigns a number to each feature of a data point: a large number marks a feature which strongly affects the prediction, while a number near zero marks a feature that barely affects it.

Machine learning methods compute feature importance scores for different purposes. As an explanation in explainable artificial intelligence, a score describes how much a learned hypothesis relies on individual features of a data point (Molnar, 2025). During model selection, importance scores can guide the choice of a subset of the features of the data points, after which a new hypothesis is learned using that subset (Guyon and Elisseeff, 2003).

![Figure 1 of the entry featureimportance](https://dictionaryofml.org/terms/fig/featureimportance-fig1.svg)

*Figure 1: Measured maximum daytime temperature (gray stems with circular markers) at the weather station Krems an der Donau (Austria) for June 2024, and the corresponding predictions (black crosses) of a hypothesis learned by linear regression from the twelve features of the two preceding days. Data generated by pythondemos/featureimportance.py*

A natural choice of feature importance scores for a learned linear model $\widehat{h}({\mathbf x}) =
\sum_{j=1}^{12} \widehat{w}_{j}
x_{j}$ are the weights $\widehat{w}_{j}$. Increasing the $j$-th feature of a data point by one unit changes the prediction by $\widehat{w}_{j}$. A feature $x_{j}$ with a large magnitude $|\widehat{w}_{j}|$ can then be considered important.

One challenge of using the weights $\widehat{w}_{j}$ delivered by linear regression as feature importance scores is their dependence on the scale of the feature values. Scaling one feature by $10^{-5}$ makes linear regression learn a weight $\widehat{w}_{j}$ that is larger by the factor $10^{5}$, while every prediction stays the same. By rescaling a feature, or changing its measurement unit, its importance can therefore be made arbitrarily large or small. This dependence is avoided by applying linear regression to standardized features (see feature standardization): each column of the feature matrix is centered by its sample mean and divided by its standard deviation, so that $\widehat{w}_{j}$ becomes the prediction change per standard deviation of the $j$-th feature.

Algorithm 1 summarizes the construction of feature importance scores by applying linear regression to the standardized feature values of a given dataset with feature matrix ${\mathbf X} \in
\mathbb{R}^{m \times d}$, whose rows are the feature vectors $\big({\mathbf x}^{(r)}\big)^{\top}$, and label vector ${\mathbf y} = \big(y^{(1)}, \ldots,
y^{(m)}\big)^{\top}$. Algorithm 1 can also be used to compute feature importance scores for a given learned hypothesis (or trained model) $\widehat{h}$ that might be nonlinear (Ribeiro et al., 2016). Here, the label vector fed into Algorithm 1 contains the predictions of $\widehat{h}$, ${\mathbf y} = \big(
\widehat{h}({\mathbf x}^{(1)}), \ldots,
\widehat{h}({\mathbf x}^{(m)}) \big)^{\top}$, so the weights are those of a linear approximation of $\widehat{h}$ on the given dataset.

![Algorithm 1 of the entry featureimportance](https://dictionaryofml.org/terms/fig/featureimportance-alg1.svg)

*Algorithm 1: Feature importance from linear regression on standardized features*

Another construction of a feature importance score starts from a given hypothesis $\widehat{h}$ delivered by some machine learning method, and scores that hypothesis instead of learning one. This construction, introduced for random forests under the name variable importance (Breiman, 2001, Sect. 10), measures the change of the average loss when the values of a single feature are replaced by random values. In the original version, each decision tree of the random forest is scored on the training data points that were not in its own resampled training set, and the score is the relative increase of the misclassification rate. Algorithm 2 summarizes the method in a form that applies to a learned hypothesis of any kind. Fisher et al. (2019, Sect. 3) introduce this form under the name model reliance, for a fixed hypothesis: the ratio, or alternatively the difference, of the expected loss with the $j$-th feature replaced by an independent copy and the expected loss without the replacement. They estimate the loss after the replacement over all pairs of data points instead of by a random permutation, and show that the score can be estimated on the training set itself. Algorithm 2 uses the difference and $B$ random permutations, as the original construction does. Its input is a dataset, characterized by a feature matrix ${\mathbf X}
\in \mathbb{R}^{m \times d}$ and a label vector ${\mathbf y} \in \mathbb{R}^{m}$, and a learned hypothesis (or trained model) $\widehat{h}$. This dataset is referred to as the scoring dataset. It should have the same statistical properties as the training set used to learn $\widehat{h}$: ideally, the data points of both are realizations of independent and identically distributed RVs.

The score $I_{j}$ of the $j$-th feature compares the average loss of $\widehat{h}$ on the dataset $\big({\mathbf X}, {\mathbf y}\big)$ with its average loss on a perturbed dataset with the same ${\mathbf y}$ but a different feature matrix $\widetilde{{\mathbf X}}^{(j)} =
\big(\widetilde{{\mathbf x}}^{(1)}, \ldots,
\widetilde{{\mathbf x}}^{(m)}\big)^{\top}$. The perturbed feature matrix is obtained by permuting the entries of the $j$-th column of ${\mathbf X}$ across its rows, with every other column unchanged (Fig. 2). The permutation importance of the $j$-th feature is the increase of the average loss that this exchange causes,

$$
I_{j} :=
\frac{1}{m} \sum_{r=1}^{m}
L\left(\left( \widetilde{{\mathbf x}}^{(r)},y^{(r)} \right),\widehat{h} \right)
- \frac{1}{m} \sum_{r=1}^{m}
L\left(\left( {\mathbf x}^{(r)},y^{(r)} \right),\widehat{h} \right)
\text{,}
$$

where $y^{(r)}$ is the label of data point $r$. The permutation is random, so Algorithm 2 averages $I_{j}$ over $B$ independent permutations of the column.

![Algorithm 2 of the entry featureimportance](https://dictionaryofml.org/terms/fig/featureimportance-alg2.svg)

*Algorithm 2: Permutation importance (variable importance), similar to Fisher et al. (2019, Sect. 3)*

![Figure 2 of the entry featureimportance](https://dictionaryofml.org/terms/fig/featureimportance-fig2.svg)

*Figure 2: Left: the feature matrix ${\mathbf X}$ of a dataset provided to Algorithm 2. This dataset is obtained from daily weather observations at Krems an der Donau; the shaded column holds the maximum temperature of the day before. The first row is the data point of October 1, 2024, whose first feature, $15.4$, is the maximum temperature of September 30. Right: the perturbed feature matrix $\widetilde{{\mathbf X}}^{(1)}$ obtained by permuting the entries of that column across the $92$ rows while every other column stays as it is. The permutation importance of the feature is the increase of the average loss when the predictions are computed from the rows of $\widetilde{{\mathbf X}}^{(1)}$ instead of the rows of ${\mathbf X}$. Values from pythondemos/featureimportance_weather.csv; the permutation is the first one drawn by pythondemos/featureimportance.py*

Fig. 3 plots $I_{j}$ computed by Algorithm 2 with $B = 20$ for the weather data points of Krems an der Donau. The scoring dataset consists of the $92$ data points whose label is the maximum temperature of a day from October 1 to December 31, 2024, so their features are the observations of September 29 to December 30. The linear hypothesis of Fig. 1 was learned on the training set of the $272$ data points whose label days run from January 3 to September 30. The maximum temperature of the day before obtains the largest score, while the minimum temperature of the day before obtains a score near zero. That minimum temperature is not useless: its correlation with the label on the scoring dataset is $0.69$, as strong as that of the minimum temperature two days before. The score $I_{j}$ primarily reflects how much $\widehat{h}$ relies on the feature $x_{j}$, which can be different from its relevance for the label of a data point.

![Figure 3 of the entry featureimportance](https://dictionaryofml.org/terms/fig/featureimportance-fig3.svg)

*Figure 3: Permutation-importance scores (Algorithm 2, $B = 20$) for the twelve features of the data points used for the temperature forecast of Fig. 1, in ascending order of $I_{j}$ (black crosses), and the value $2 \widehat{w}_{j}^{2} \operatorname{var}\left(x_{j}\right)$ computed from the weights of the linear hypothesis (gray circles), which the squared error loss derivation below gives as an approximation of $I_{j}$. Algorithm 2 was run on the scoring dataset, the $92$ data points whose label days run from October 1 to December 31, 2024, whereas the linear hypothesis was learned on the data points whose label days run from January 3 to September 30. Tick labels: max temp and min temp are the maximum and the minimum air temperature, precip the precipitation, and day before and two days before count back from the day whose maximum temperature is the label. Data generated by pythondemos/featureimportance.py*

Algorithm 2 requires a choice of the loss function as input. For the squared error loss, the scores delivered by Algorithm 2 are closely related to the coefficients delivered by Algorithm 1. Indeed, randomly permuting column $j$ of ${\mathbf X}$ is the same as replacing $x_{j}$ with the realization of a random variable $x_{j}^{\prime}$. For a linear model, this replacement changes the prediction by $\delta_{j} =
\widehat{w}_{j} (x_{j}^{\prime}
- x_{j})$. The squared error loss increase is

\begin{equation}
\tag{1}
\delta_{j}^{2} - 2 \big(y -
\widehat{h}({\mathbf x})\big) \delta_{j}
\text{,}
\end{equation}

where $y$ is the label of the data point and $\widehat{h}({\mathbf x})$ the prediction for its feature vector. The score $I_{j}$ delivered by Algorithm 2 is obtained by averaging the loss increase Eq. (1) over all data points of the scoring dataset. The random variable $x_{j}^{\prime}$ has the same variance as $x_{j}$, and if the two have zero covariance, then $\mathbb{E}\{(x_{j}^{\prime}
-x_{j})^{2}\}
=2\,\operatorname{var}\left(x_{j}\right)$. Suppose that the prediction error $y-\widehat{h}({\mathbf x})$ has expectation zero given ${\mathbf x}$, as it does when $\widehat{h}$ is the conditional mean of $y$ given ${\mathbf x}$, and that $x_{j}^{\prime}$ is independent of the data point. Then this error is uncorrelated with $\delta_{j}$, the second term of the loss increase Eq. (1) has expectation zero, and the expected loss increase is $\mathbb{E}\{\delta_{j}^{2}\} =
2\widehat{w}_{j}^{2}
\operatorname{var}\left(x_{j}\right)$ (Gregorutti et al., 2017, Prop. 1). A learned $\widehat{h}$ only approximates that conditional mean, and a permuted column only approximates an independent copy, so the score satisfies $I_{j} \approx
2\widehat{w}_{j}^{2}
\operatorname{var}\left(x_{j}\right)$. For standardized features, $\operatorname{var}\left(x_{j}\right) = 1$ and $I_{j} \approx 2 \widehat{w}_{j}^{2}$: for a linear hypothesis, the score of Algorithm 2 is, up to the square, the score of Algorithm 1. The relation requires that the weights are standardized with respect to the dataset on which Algorithm 2 evaluates the average loss; this dataset need not be the training set of the hypothesis. For the maximum temperature of the day before, the weight of the hypothesis of Fig. 1 times the standard deviation of that feature on the scoring dataset is $4.96$, and $\sqrt{I_{j}/2} = 4.96$.

Algorithm 2 uses $\widehat{h}$ only through its predictions, so it applies to any learned hypothesis. Fig. 4 shows the scores for a random forest, an ensemble of decision trees, learned on the same training set and evaluated on the same scoring dataset as the linear hypothesis of Fig. 3. Both reach almost the same average squared error loss on the scoring dataset, $8.2$ in squared degrees Celsius, but they rely on the features differently.

![Figure 4 of the entry featureimportance](https://dictionaryofml.org/terms/fig/featureimportance-fig4.svg)

*Figure 4: Importance scores obtained from Algorithm 2 ($B = 20$; black circles) for the weather dataset of Fig. 1, for a random forest learned on the training set with label days from January 3 to September 30, 2024, and scored on the scoring dataset of Fig. 3, with label days from October 1 to December 31, in the feature order of that figure so that the two can be compared column by column. Tick labels as in Fig. 3. Data generated by pythondemos/featureimportance.py*

**Synonyms:** variable importance.

**See also:** [feature](https://dictionaryofml.org/terms/feature.html), [feature matrix](https://dictionaryofml.org/terms/featuremtx.html), linear model, [loss](https://dictionaryofml.org/terms/loss.html), [random forest](https://dictionaryofml.org/terms/randomforest.html), local interpretable model-agnostic explanations, covariance.

## References

- Fisher et al. (2019). All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously. *Journal of Machine Learning Research*. <https://www.jmlr.org/papers/v20/18-760.html>
- Molnar (2025). Interpretable Machine Learning: A Guide for Making Black Box Models Explainable. *Ebook*. <https://christophm.github.io/interpretable-ml-book/>
- Guyon and Elisseeff (2003). An introduction to variable and feature selection. *Journal of machine learning research*.
- Ribeiro et al. (2016). ``Why Should I Trust You?'': Explaining the Predictions of Any Classifier. *Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining*. <https://doi.org/10.1145/2939672.2939778>
- Breiman (2001). Random Forests. *Machine Learning*. <https://doi.org/10.1023/A:1010933404324>
- Gregorutti et al. (2017). Correlation and variable importance in random forests. *Statistics and Computing*. <https://doi.org/10.1007/s11222-016-9646-1>

## Cite this entry

```bibtex
@misc{dictml_featureimportance,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {feature importance},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/featureimportance.html}
}
```


---

# feature matrix

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-11. [HTML](https://dictionaryofml.org/terms/featuremtx.html) · [typeset PDF](https://dictionaryofml.org/terms/featuremtx.pdf) · [LaTeX](https://dictionaryofml.org/terms/featuremtx.tex) · [Python demo](https://dictionaryofml.org/terms/featuremtx.py) · [notebook](https://dictionaryofml.org/terms/featuremtx.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

The feature matrix of a dataset collects the feature vectors of its data points as its rows, one row per data point and one column per feature. Its two most basic properties are the number of rows, which is the number of data points in the dataset, and the number of columns, which is the number of features of each data point. More properties of a dataset follow from the singular values of its feature matrix. The number of nonzero singular values is the rank, and a rank below the number of features leaves linear regression with infinitely many solutions. The ratio of the largest to the smallest nonzero singular value is the condition number of the feature matrix. The condition number determines the number of gradient descent steps required by linear regression and its robustness against perturbations of the features and labels in the training set.

## Definition

A weather station at Krems an der Donau records eight measurements on each of the $366$ days of the leap year $2024$: the lowest, highest, and mean air temperature, the humidity, the wind speed, the precipitation, the sunshine duration, and the air pressure. Writing the eight measurements of one day as a row, and placing the rows of the $366$ days under one another, gives a two-dimensional numeric array with $366$ rows and eight columns. That array is a matrix, the feature matrix of the dataset whose data points are the $366$ days, and each of its rows is the feature vector of one day. The feature matrix is the form in which machine learning software expects a dataset, and properties of the machine learning methods applied to the dataset, such as whether linear regression has a unique solution or how fast gradient descent converges, are determined by it.

![Figure 1 of the entry featuremtx](https://dictionaryofml.org/terms/fig/featuremtx-fig1.svg)

*Figure 1: The feature matrix of the $366$ days of $2024$ at Krems an der Donau: one row per day and one column per measurement, with the first three days and the last shown, and five of the eight measurements (the precipitation, the sunshine duration, and the air pressure are omitted). Temperatures are in degrees Celsius, the humidity in percent, and the wind speed in meters per second. The shaded row is the transposed feature vector of 2 January; the heavily framed column collects the lowest temperature of every day*

In general, consider a dataset $\mathcal{D}$ of $m$ data points with feature vectors ${\mathbf x}^{(1)}, \,\ldots, \,{\mathbf x}^{(m)}$, each in $\mathbb{R}^{d}$. Stacking the feature vectors row-wise yields the feature matrix:

$$
{\mathbf X} := \big({\mathbf x}^{(1)}, \,\ldots, \,{\mathbf x}^{(m)}\big)^{\top} =
\begin{pmatrix}
x^{(1)}_{1} & x^{(1)}_{2} & \cdots & x^{(1)}_{d} \\
x^{(2)}_{1} & x^{(2)}_{2} & \cdots & x^{(2)}_{d} \\
\vdots & \vdots & \ddots & \vdots \\
x^{(m)}_{1} & x^{(m)}_{2} & \cdots & x^{(m)}_{d}
\end{pmatrix}
\in \mathbb{R}^{m \times d} \text{.}
$$

The feature matrix is of size $m \times d$, i.e., it has $m$ rows and $d$ columns. Each row is the transposed feature vector of one data point, and each column holds the values of one feature for all data points (see Fig. 1). Many properties of a dataset, and of the learning tasks built from it, are computed from ${\mathbf X}$. The most basic ones are the numbers of rows and columns, $m$ and $d$.

The feature matrix also defines a linear map. A linear model with model parameters ${\mathbf w}$ from the parameter space $\mathbb{R}^{d}$ predicts ${\mathbf w}^{\top} {\mathbf x}^{(r)}$ for the $r$-th data point, so the $m$ predictions are the entries of ${\mathbf X} {\mathbf w}$. The feature matrix therefore maps each ${\mathbf w} \in \mathbb{R}^{d}$ to the vector ${\mathbf X} {\mathbf w} \in \mathbb{R}^{m}$, which is compared with the label vector ${\mathbf y} = \big(y^{(1)}, \,\ldots,
\,y^{(m)}\big)^{\top} \in \mathbb{R}^{m}$. Linear regression on a training set with feature matrix ${\mathbf X}$ and label vector ${\mathbf y}$ is the search for a vector ${\mathbf w}$ whose image ${\mathbf X} {\mathbf w}$ is close to ${\mathbf y}$ in Euclidean norm.

Linear regression has a unique solution only when ${\mathbf X}$ has full column rank, which requires $m \geq d$ (Golub and Loan, 2013, Sect. 5.3.1) (see normal equations). A rank below $d$ leaves infinitely many parameter vectors ${\mathbf w}$ with the same training error (Hastie et al., 2009, Sect. 3.2), and the one a method returns can have a large loss outside the training set. Using more features than data points (i.e., $d > m$) forces this case, since then ${\mathbf X}$ has fewer rows than columns, which rules out full column rank.

More properties of a dataset follow from the singular values of its feature matrix ${\mathbf X}$. The singular value decomposition can be written as (Golub and Loan, 2013, Sect. 2.4)

$$
{\mathbf X} = \sum_{j=1}^{\min\{m, d\}}
\lambda_{j} {\mathbf v}^{(j)}
\big({\mathbf u}^{(j)}\big)^{\top} \text{,}
$$

where the vectors ${\mathbf v}^{(1)}, \,\ldots, \,{\mathbf v}^{(\min\{m, d\})} \in \mathbb{R}^{m}$ are orthonormal among themselves, the vectors ${\mathbf u}^{(1)}, \,\ldots, \,{\mathbf u}^{(\min\{m, d\})} \in \mathbb{R}^{d}$ are orthonormal among themselves. The singular values are ordered as $\lambda_{1} \geq \ldots \geq \lambda_{\min\{m, d\}} \geq 0$. The number of nonzero singular values is the rank of ${\mathbf X}$ (Golub and Loan, 2013, Cor. 2.4.6). A rank below $d$ means that some feature is a linear combination of the others and adds no information beyond them. A small but nonzero singular value is the approximate form of such a dependence. When $m \geq d$, the $d$ right singular vectors form an orthonormal basis of $\mathbb{R}^{d}$, and the smallest singular value is the smallest Euclidean norm of a linear combination of the feature columns with unit-norm coefficients,

$$
\lambda_{d}
= \min_{{\mathbf u} \in \mathbb{R}^{d}, \, \Vert {{\mathbf u}} \Vert = 1}
\Vert {{\mathbf X} {\mathbf u}} \Vert \text{,}
$$

attained at the right singular vector ${\mathbf u}^{(d)}$. This holds since a unit-norm ${\mathbf u} = \sum_{j=1}^{d} c_{j}
{\mathbf u}^{(j)}$ gives $\Vert {{\mathbf X} {\mathbf u}} \Vert^{2} =
\sum_{j=1}^{d} c_{j}^{2} \lambda_{j}^{2}$, a weighted average of the squared singular values. When $d > m$, a unit-norm ${\mathbf u}$ orthogonal to all $m$ right singular vectors gives ${\mathbf X} {\mathbf u} = \mathbf{0}$, so the minimum is $0$. A small singular value therefore says that some combination of the feature columns with unit-norm coefficients is nearly the zero vector: the features are nearly collinear, and the right singular vector names the combination. For the weather year of Fig. 1, with every measurement scaled to zero sample mean and unit sample variance, the smallest singular value is $0.05$ against a largest of $36.4$. Its right singular vector places all of its weight on the lowest, the highest, and the mean temperature, with coefficients $-0.37$, $-0.46$, and $0.81$, because the mean temperature of a day is the midpoint of its lowest and its highest up to a residual with standard deviation $0.03$ degrees. Dropping the mean temperature raises the smallest singular value to $4.19$ (pythondemos/featuremtx.py). The size of a singular value depends on the units of the columns, which is why the scaling to unit sample variance comes first. The squares of the singular values are the eigenvalues of ${\mathbf X}^{\top} {\mathbf X}$, which for features with zero sample mean is $m$ times their sample covariance matrix.

The same eigenvalues and eigenvectors of ${\mathbf X}^{\top} {\mathbf X}$ determine principal component analysis. Its dimensionality reduction keeps the eigenvectors of the largest eigenvalues as the new features, and the sum over the data points of the squared errors of reconstructing the feature vectors from them is the sum of the eigenvalues dropped (Bishop, 2006, Sect. 12.1.2).

Another important property of a feature matrix is the condition number $\kappa\!\left({\mathbf X}\right) = \lambda_{1} / \lambda_{\operatorname{rank}({\mathbf X})}$, the ratio of the largest singular value to the smallest nonzero one. For full column rank, it is the ratio of the largest to the smallest singular value (Golub and Loan, 2013, Sect. 5.3.1). The condition number enters linear regression twice: through the number of gradient descent steps, and through the robustness of the learned model parameters against perturbations of the training set. The number that matters in both is its square $\kappa\!\left({\mathbf X}\right)^{2}$. The objective function $f({\mathbf w}) = (1/m) \Vert {{\mathbf X} {\mathbf w} -
{\mathbf y}} \Vert^{2}$ has the constant Hessian $(2/m) {\mathbf X}^{\top} {\mathbf X}$, whose eigenvalues are the squared singular values of ${\mathbf X}$ scaled by $2/m$. The condition number of ${\mathbf X}^{\top} {\mathbf X}$ is therefore $\kappa\!\left({\mathbf X}\right)^{2}$.

The square $\kappa\!\left({\mathbf X}\right)^{2}$ determines the convergence speed of gradient descent on the empirical risk minimization problem of linear regression. One step with an exact line search multiplies the gap between $f\big({\mathbf w}^{(t)}\big)$ and the minimum $f\big(\widehat{{\mathbf w}}\big)$ by at most $1 - 1/\kappa\!\left({\mathbf X}\right)^{2}$ (Boyd and Vandenberghe, 2004, Sect. 9.3.1). The number of steps that reaches a given gap therefore grows in proportion to $\kappa\!\left({\mathbf X}\right)^{2}$.

Fig. 2 shows this for two feature matrices built from the weather measurements of Fig. 1. The features are the relative humidity and the wind speed, the label is the highest temperature, and each has its sample mean removed. In the recorded units the condition number is $14.5$. After scaling each feature to unit sample variance, it is $1.45$. Gradient descent starts at the point at distance $\Vert {\widehat{{\mathbf w}}} \Vert$ from $\widehat{{\mathbf w}}$ for which it is slowest, where the gap shrinks by exactly $\big((\kappa\!\left({\mathbf X}\right)^{2} - 1) /
(\kappa\!\left({\mathbf X}\right)^{2} + 1)\big)^{2}$ per step (Boyd and Vandenberghe, 2004, Sect. 9.3.2); from no start does the gap shrink by less, since this factor bounds the shrinkage of every exact-line-search step on a quadratic objective function (Bertsekas, 2016, Prop. 1.3.1). A gap of $10^{-6}$ of the initial one then takes $731$ steps in the recorded units and $7$ after the scaling.

![Figure 2 of the entry featuremtx](https://dictionaryofml.org/terms/fig/featuremtx-fig2.svg)

*Figure 2: The convergence of gradient descent with an exact line search for the features relative humidity and wind speed of the $366$ days, in their recorded units (solid curve) and after scaling each to unit sample variance (dashed curve). Shown is the gap between $f\big({\mathbf w}^{(t)}\big)$ and the minimum $f^{\star} = \min_{{\mathbf w} \in \mathbb{R}^{2}} f({\mathbf w})$, relative to its initial value, over the first $50$ steps. Both runs start at the point at distance $\Vert {\widehat{{\mathbf w}}} \Vert$ from $\widehat{{\mathbf w}}$ for which gradient descent is slowest; the dashed curve ends where the gap falls below $10^{-12}$. Data generated by pythondemos/featuremtx.py*

The condition number $\kappa\!\left({\mathbf X}\right)$ also governs the robustness of the model parameters $\widehat{{\mathbf w}}$ delivered by linear regression against perturbations of the training set. Let ${\mathbf X}$ have full column rank. Linear regression delivers the model parameters $\widehat{{\mathbf w}} = {{\mathbf X}}^{+} {\mathbf y}$ with the pseudoinverse ${{\mathbf X}}^{+} = \big({\mathbf X}^{\top} {\mathbf X}\big)^{-1}
{\mathbf X}^{\top}$ (Golub and Loan, 2013, Sect. 5.5.2). They are the unique solution of the normal equations ${\mathbf X}^{\top} {\mathbf X} \widehat{{\mathbf w}} =
{\mathbf X}^{\top} {\mathbf y}$, which state that the gradient of $f$ vanishes at $\widehat{{\mathbf w}}$.

Consider perturbations of the training set that change ${\mathbf X}$ to ${\mathbf X} + \delta {\mathbf X}$ and ${\mathbf y}$ to ${\mathbf y} + \delta {\mathbf y}$, with ${\mathbf X} + \delta {\mathbf X}$ still of full column rank, and let $\widehat{{\mathbf w}} + \delta \widehat{{\mathbf w}}$ be the model parameters learned by linear regression on the perturbed training set. They solve the normal equations of the perturbed training set,

$$
({\mathbf X} + \delta {\mathbf X})^{\top}
({\mathbf X} + \delta {\mathbf X})
\big(\widehat{{\mathbf w}} + \delta \widehat{{\mathbf w}}\big)
= ({\mathbf X} + \delta {\mathbf X})^{\top}
({\mathbf y} + \delta {\mathbf y}) \text{.}
$$

Moving the terms with $\widehat{{\mathbf w}}$ to the right side and using ${\mathbf X}^{\top} {\mathbf r} = \mathbf{0}$ for the least squares residual ${\mathbf r} = {\mathbf y} - {\mathbf X} \widehat{{\mathbf w}}$, which is the normal equations of the unperturbed training set, gives the exact equation

$$
({\mathbf X} + \delta {\mathbf X})^{\top}
({\mathbf X} + \delta {\mathbf X}) \, \delta \widehat{{\mathbf w}}
= ({\mathbf X} + \delta {\mathbf X})^{\top}
\big(\delta {\mathbf y} - \delta {\mathbf X} \, \widehat{{\mathbf w}}\big)
+ \delta {\mathbf X}^{\top} {\mathbf r} \text{.}
$$

Solving for $\delta \widehat{{\mathbf w}}$ applies the pseudoinverse of ${\mathbf X} + \delta {\mathbf X}$ to the noise in the labels and to the noise in the features acting on $\widehat{{\mathbf w}}$, and the inverse of $({\mathbf X} + \delta {\mathbf X})^{\top}
({\mathbf X} + \delta {\mathbf X})$ to the noise in the features acting on the residual.

The largest singular value of ${{\mathbf X}}^{+}$ is $1 / \lambda_{d}$, and that of $\big({\mathbf X}^{\top} {\mathbf X}\big)^{-1}$ is $1 / \lambda_{d}^{2}$ (Golub and Loan, 2013, Sect. 5.3.6). The same holds for ${\mathbf X} + \delta {\mathbf X}$ with its own smallest singular value. Let the largest singular value of $\delta {\mathbf X}$ be at most $\varepsilon \lambda_{1}$ and the Euclidean norm of $\delta {\mathbf y}$ at most $\varepsilon \Vert {{\mathbf y}} \Vert$, with $\varepsilon \, \kappa\!\left({\mathbf X}\right) < 1$. The smallest singular value of ${\mathbf X} + \delta {\mathbf X}$ is then at least $\lambda_{d} - \varepsilon \lambda_{1}$ (Golub and Loan, 2013, Cor. 8.6.2), and taking Euclidean norms in the exact equation gives the perturbation bound

\begin{equation}
\tag{1}
\Vert {\delta \widehat{{\mathbf w}}} \Vert \leq
\frac{\varepsilon \, \kappa\!\left({\mathbf X}\right)}
{1 - \varepsilon \, \kappa\!\left({\mathbf X}\right)}
\Big( \frac{\Vert {{\mathbf y}} \Vert}{\lambda_{1}}
+ \Vert {\widehat{{\mathbf w}}} \Vert \Big)
+ \frac{\varepsilon \, \kappa\!\left({\mathbf X}\right)^{2}}
{\big(1 - \varepsilon \, \kappa\!\left({\mathbf X}\right)\big)^{2}}
\frac{\Vert {{\mathbf r}} \Vert}{\lambda_{1}} \text{.}
\end{equation}

The bound Eq. (1) holds for every perturbation with $\varepsilon \, \kappa\!\left({\mathbf X}\right) < 1$.

The bound Eq. (1) carries over to the prediction for any given feature vector ${\mathbf x}$, such as that of a test data point. The learned linear model predicts ${\mathbf x}^{\top} \widehat{{\mathbf w}}$ before and ${\mathbf x}^{\top} \big(\widehat{{\mathbf w}} + \delta \widehat{{\mathbf w}}\big)$ after the perturbation. The prediction therefore changes by ${\mathbf x}^{\top} \delta \widehat{{\mathbf w}}$, and the Cauchy-Schwarz inequality bounds this change by

\begin{equation}
\tag{2}
\big| {\mathbf x}^{\top} \delta \widehat{{\mathbf w}} \big|
\leq \Vert {{\mathbf x}} \Vert \, \Vert {\delta \widehat{{\mathbf w}}} \Vert
\leq \Vert {{\mathbf x}} \Vert \Bigg(
\frac{\varepsilon \, \kappa\!\left({\mathbf X}\right)}
{1 - \varepsilon \, \kappa\!\left({\mathbf X}\right)}
\Big( \frac{\Vert {{\mathbf y}} \Vert}{\lambda_{1}}
+ \Vert {\widehat{{\mathbf w}}} \Vert \Big)
+ \frac{\varepsilon \, \kappa\!\left({\mathbf X}\right)^{2}}
{\big(1 - \varepsilon \, \kappa\!\left({\mathbf X}\right)\big)^{2}}
\frac{\Vert {{\mathbf r}} \Vert}{\lambda_{1}} \Bigg) \text{.}
\end{equation}

The change of the prediction is therefore at most $\Vert {{\mathbf x}} \Vert$ times the right-hand side of the bound Eq. (1), and grows with the norm of the feature vector.

Fig. 3 shows the robustness of $\widehat{{\mathbf w}}$ against perturbations on a related but different and smaller dataset, the first days of the twelve months of $2024$, twelve of the rows of Fig. 1. Each day has the feature vector ${\mathbf x} = (1, x)^{\top}$ with $x$ its lowest temperature, and its highest temperature as the label. A hypothesis of the linear model then has intercept $w_{1}$ and slope $w_{2}$, and the feature matrix has a column of ones and the column of the lowest temperatures. Its condition number is $14.9$, and the least squares residual is $0.20$ times the norm of ${\mathbf y}$.

Two constructions of a perturbation of relative size $\varepsilon = 0.01$ are compared. The first is chosen adversarially, to make the change of the model parameters learned by linear regression as large as possible: $\delta {\mathbf X} = -\varepsilon \lambda_{1} {\mathbf v}^{(2)} \big({\mathbf u}^{(2)}\big)^{\top}$ and $\delta {\mathbf y} = \pm \varepsilon \Vert {{\mathbf y}} \Vert {\mathbf v}^{(2)}$, with the sign for which $\widehat{{\mathbf w}}$ moves more. It changes $\widehat{{\mathbf w}}$ by norm $2.23$, which is $64$ percent of the right-hand side of the bound Eq. (1). The second construction uses $100$ realizations of random $\delta {\mathbf X}$ and $\delta {\mathbf y}$. They are generated with independent and identically distributed Gaussian RVs as their entries and then normalized so that the largest singular value of $\delta {\mathbf X}$ is $\varepsilon \lambda_{1}$ and the Euclidean norm of $\delta {\mathbf y}$ is $\varepsilon \Vert {{\mathbf y}} \Vert$. The largest change among the $100$ realizations is $1.17$, which is $34$ percent of the right-hand side of the bound Eq. (1).

For a day with lowest temperature $x$ the feature vector $(1, x)^{\top}$ has norm $\sqrt{1 + x^{2}}$. By the prediction bound Eq. (2) the prediction therefore changes by at most $\sqrt{1 + x^{2}}$ times the right-hand side of the bound Eq. (1): $3.5$ degrees at $x = 0$ and $69$ degrees at $x = 20$. These are the two dotted curves in Fig. 3(a), which leave the plotted range. The hypotheses learned from the random perturbations stay within $1.6$ degrees of the learned hypothesis over the plotted range, and the hypothesis after the adversarial construction within $3.1$ degrees. Panel (b) repeats the comparison for $\varepsilon$ from $5 \cdot 10^{-4}$ to $5 \cdot 10^{-2}$. The adversarial construction reaches more than half of the right-hand side of the bound Eq. (1) up to $\varepsilon = 0.02$, the random construction less than half at every $\varepsilon$.

![Figure 3 of the entry featuremtx](https://dictionaryofml.org/terms/fig/featuremtx-fig3.svg)

*Figure 3: The perturbation bound for the features $(1, x)$, with $x$ the lowest temperature of a day, and the highest temperature as the label, over the first days of the twelve months of $2024$ (filled circles). (a) The least squares hypothesis (solid line), the hypothesis learned after the adversarial construction of a perturbation of relative size $\varepsilon = 0.01$ (dashed line), the range of the hypotheses learned after $100$ random constructions of the same relative size (gray band), and the bound Eq. (2) above and below the learned hypothesis (dotted curves, leaving the plotted range). (b) The change of the learned model parameters against $\varepsilon$: the right-hand side of the bound Eq. (1) (solid curve), the adversarial construction (dashed curve), and the largest change among $100$ random constructions (dotted curve with circles). Data generated by pythondemos/featuremtx.py*

**Synonyms:** design matrix, data matrix.

**See also:** [dataset](https://dictionaryofml.org/terms/dataset.html), [data point](https://dictionaryofml.org/terms/datapoint.html), [feature vector](https://dictionaryofml.org/terms/featurevec.html), [feature](https://dictionaryofml.org/terms/feature.html), array, [matrix](https://dictionaryofml.org/terms/matrix.html), design matrix, linear model, parameter space, [singular value decomposition](https://dictionaryofml.org/terms/svd.html), singular value, rank, condition number, sample covariance matrix, [principal component analysis](https://dictionaryofml.org/terms/pca.html), [dimensionality reduction](https://dictionaryofml.org/terms/dimred.html), column space.

## References

- Golub and Loan (2013). Matrix Computations. *The Johns Hopkins Univ. Press*.
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Boyd and Vandenberghe (2004). Convex Optimization. *Cambridge Univ. Press*. <https://doi.org/10.1017/CBO9780511804441>
- Bertsekas (2016). Nonlinear Programming. *Athena Scientific*.

## Cite this entry

```bibtex
@misc{dictml_featuremtx,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {feature matrix},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/featuremtx.html}
}
```


---

# feature vector

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/featurevec.html) · [typeset PDF](https://dictionaryofml.org/terms/featurevec.pdf) · [LaTeX](https://dictionaryofml.org/terms/featurevec.tex) · [Python demo](https://dictionaryofml.org/terms/featurevec.py) · [notebook](https://dictionaryofml.org/terms/featurevec.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A feature vector is a list of numbers describing a data point, with one entry for each of its features. A feature is an attribute of the data point that is measured or computed without human supervision, so the length of the vector is the number of such attributes. A feature transformation computes the feature vector from the data point and delivers a point of the feature space, to which a hypothesis map is then applied to predict the label. Non-numeric features enter as numbers: a binary attribute as $0$ or $1$, a categorical one through one-hot encoding. Most machine learning methods use feature vectors in a Euclidean space whose dimension is the number of features. A feature vector can more generally be an element of a Hilbert space, which is how a kernel method uses one without ever forming it, using it only through inner products.

## Definition

A house offered for sale can be described to a machine learning method by three numbers: $85$ square meters of living area, $3$ rooms, and $1974$ as the year of construction. That list $\big(85, 3, 1974\big)$ is the feature vector of the house. In general, a feature vector gathers into one vector the features of a data point, those attributes of the data point that are measured or computed without human supervision (Duda et al., 2001). A list of $d$ real numbers, for a natural number $d$, is an element of the Euclidean space $\mathbb{R}^{d}$ (Strang, 2016, Sect. 1.1).

The feature vector of a data point ${\mathbf z}$ is written ${\mathbf x} = \big(x_{1}, \,\ldots,
\,x_{d}\big)^{\top}$, and its entry $x_{j}$ is one feature of that data point. It is obtained from the data point by a feature transformation ${\mathbf \Phi} : \mathcal{Z} \rightarrow \mathcal{X}$, ${\mathbf x} = {\mathbf \Phi}({\mathbf z})$, which views the data point itself as its raw features and delivers a point of the feature space $\mathcal{X}$ (Fig. 1). A hypothesis map takes the feature vector as input and outputs a prediction of the label of the data point.

![Figure 1 of the entry featurevec](https://dictionaryofml.org/terms/fig/featurevec-fig1.svg)

*Figure 1: A feature vector ${\mathbf x}$ stacks the individual features $x_{1}, \,\ldots, \,x_{d}$ of a data point ${\mathbf z}$, computed by the feature transformation ${\mathbf \Phi}$. It is a point in the feature space $\mathcal{X}$*

Non-numeric features can also sit in the entries of a feature vector, once their values are encoded as real numbers (Hastie et al., 2009; Goodfellow et al., 2016). A binary feature, such as whether the house has a balcony, becomes $0$ or $1$. A categorical feature with $K$ possible values, such as the heating type (gas, oil, or electric), is one-hot encoded: $K$ entries of the feature vector carry the value $0$ except for a single entry equal to $1$, whose position names the value.

A feature transformation is also defined for data points that are not records of numbers. A text becomes a feature vector by counting how often each entry of a fixed word list appears in it, which gives one entry per listed word. A digital image is an array of pixels, and the red, green and blue intensities of those pixels become one entry each. Reading a $2 \times 2$ patch of such an image this way gives $2 \cdot 2 \cdot 3 = 12$ features (Fig. 2). An audio recording becomes a feature vector through the values its signal takes at successive instants, one entry $x_{j}$ per instant $j$. The feature transformation differs in each case, and so does the dimension $d$ of the resulting feature space: the house above needs seven entries once its balcony and heating type are encoded, while an audio recording read at $256$ instants needs $256$ (Fig. 3). The order of the entries is arbitrary but fixed: every data point must be read in the same order, since a hypothesis map weights the entry at a given position, not the feature at that position in another data point.

![Figure 2 of the entry featurevec](https://dictionaryofml.org/terms/fig/featurevec-fig2.svg)

*Figure 2: A $2 \times 2$ patch of a digital image read as a feature vector. The patch carries one array of intensities per color channel, drawn as the three grids labeled R, G and B; each grid holds the same four pixels in the same positions. On the right the patch is shown in color, its pixels numbered $1$ to $4$: pixel $1$ is the color that the three intensities $78$, $85$ and $67$ make together. Stacking those three values, then those of pixel $2$, and so on gives a feature vector with $d = 12$ entries*

![Figure 3 of the entry featurevec](https://dictionaryofml.org/terms/fig/featurevec-fig3.svg)

*Figure 3: The feature vector of an audio recording read at $256$ instants. Entry $x_{j}$ is the value its signal takes at instant $j$, so this data point is a point of $\mathbb{R}^{256}$. Data generated by pythondemos/featurevec.py*

Most machine learning methods use feature vectors in a finite-dimensional Euclidean space $\mathbb{R}^{d}$, whose dimension $d$ is the number of features. More generally, a feature vector can be an element of an abstract Hilbert space $\mathcal{H}$. One way to build a feature vector of that kind starts from a kernel $k: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ on a given feature space $\mathcal{X}$. The kernel assigns to each ${\mathbf x} \in \mathcal{X}$ the function $k\big({\mathbf x},\cdot\big): \mathcal{X} \rightarrow \mathbb{R}$, whose value at ${\mathbf x}' \in \mathcal{X}$ is $k\big({\mathbf x},{\mathbf x}'\big)$. That function is itself a vector in a reproducing kernel Hilbert space $\mathcal{H}$, a Hilbert space whose elements are functions on $\mathcal{X}$ (Aronszajn, 1950, Part I, Sect. 2; Schölkopf and Smola, 2002, Sect. 2.2.2). A kernel method uses these feature vectors without ever forming them explicitly: it reaches them only through their inner products, which the reproducing property reduces to kernel evaluations, $\langle k\big({\mathbf x},\cdot\big), k\big({\mathbf x}',\cdot\big) \rangle_{\mathcal{H}}
= k\big({\mathbf x},{\mathbf x}'\big)$ (Schölkopf and Smola, 2002, Eq. (2.30)).

**Synonyms:** covariate vector, input vector, predictor vector.

**See also:** [feature](https://dictionaryofml.org/terms/feature.html), feature space, feature transformation, [data point](https://dictionaryofml.org/terms/datapoint.html), vector, Euclidean space, kernel, reproducing kernel Hilbert space, [kernel method](https://dictionaryofml.org/terms/kernelmethod.html), [Hilbert space](https://dictionaryofml.org/terms/hilbertspace.html).

## References

- Duda et al. (2001). Pattern Classification. *Wiley*.
- Strang (2016). Introduction to Linear Algebra. *Wellesley-Cambridge Press*.
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Goodfellow et al. (2016). Deep Learning. *MIT Press*.
- Aronszajn (1950). Theory of Reproducing Kernels. *Trans. Am. Math. Soc.*. <https://doi.org/10.1090/S0002-9947-1950-0051437-7>
- Schölkopf and Smola (2002). Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. *MIT Press*. <https://doi.org/10.7551/mitpress/4175.001.0001>

## Cite this entry

```bibtex
@misc{dictml_featurevec,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {feature vector},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/featurevec.html}
}
```


---

# fine-tuning

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/finetuning.html) · [typeset PDF](https://dictionaryofml.org/terms/finetuning.pdf) · [LaTeX](https://dictionaryofml.org/terms/finetuning.tex) · [Python demo](https://dictionaryofml.org/terms/finetuning.py) · [notebook](https://dictionaryofml.org/terms/finetuning.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Fine-tuning continues the training of a model on a task-specific dataset, starting from model parameters obtained by pretraining on a larger dataset. It solves an empirical risk minimization problem over a small training set under the constraint that the model parameters stay close to the pretrained ones. One form of fine-tuning runs a few gradient descent steps on the small training set, started at the pretrained model parameters. Forcing the model parameters to stay close to the pretrained ones is a form of regularization, since it limits the effective hypothesis space that is searched during fine-tuning. For a linear model, the generalization gap grows in proportion to how far the model parameters may move from the pretrained ones, which is why a small training set suffices.

## Definition

Consider daily weather at Krems an der Donau. A linear model that predicts the maximum daytime temperature from the morning minimum is trained on 360 days of one year, which yields a pretrained hypothesis. The task-specific datasets are the temperature measurements of a single month, and one task is to learn a forecaster tailored to that month. For a single month, thirty gradient descent steps on that month's 30 days adjust the model parameters, starting from those learned on the whole year (Fig. 1). Fine-tuning is this second step: it continues the training of a model on a task-specific dataset, starting from model parameters obtained by pretraining on a larger dataset (Bishop and Bishop, 2024, Sect. 6.3.4).

![Figure 1 of the entry finetuning](https://dictionaryofml.org/terms/fig/finetuning-fig1.svg)

*Figure 1: Pretraining and fine-tuning of a linear model that predicts the maximum daytime temperature from the morning minimum at Krems an der Donau. (a) Twenty of the 360 days of one year, every 18th day (gray dots), and the hypothesis $\widehat{h}^{(\mathrm{pre})}$ learned from all 360 (dashed line). (b) to (d) Every other day of January, May and November, 15 of each month's 30 (black dots; all three panels on the same temperature range), the pretrained hypothesis (dashed line), and the hypothesis obtained by thirty gradient descent steps on all 30 days of that month starting from the pretrained model parameters (solid line). Each fine-tuned hypothesis lowers the month's average squared error below that of the pretrained hypothesis while moving the model parameters by less than $1.5$. Data generated by pythondemos/finetuning.py*

Formally, let $f({\mathbf w})$ denote the average loss on the task-specific training set $\mathcal{D}^{(\rm train)} = \{{\mathbf z}^{(r)}\}_{r=1}^{m}$,

$$
f({\mathbf w}) :=
\frac{1}{m}
\sum_{r=1}^{m}
L\left({\mathbf z}^{(r)},{\mathbf w} \right)
\text{.}
$$

Fine-tuning solves the empirical risk minimization problem for $f$ not over the whole hypothesis space but under the constraint that the model parameters stay within a radius $B$ of the pretrained ones $\widehat{{\mathbf w}}^{(\mathrm{pre})}$,

$$
\widehat{{\mathbf w}}
\in \mathop{\mathrm{arg\,min}}_{{\mathbf w} \in
\overline{\mathcal{B}}_{B}(\widehat{{\mathbf w}}^{(\mathrm{pre})})}
f({\mathbf w})
\text{.}
$$

Here, $\overline{\mathcal{B}}_{B}(\widehat{{\mathbf w}}^{(\mathrm{pre})})
:= \{{\mathbf w} : \Vert {{\mathbf w} -
\widehat{{\mathbf w}}^{(\mathrm{pre})}} \Vert \leq B\}$ is the closed ball of radius $B$ around $\widehat{{\mathbf w}}^{(\mathrm{pre})}$ (see Fig. 2).

![Figure 2 of the entry finetuning](https://dictionaryofml.org/terms/fig/finetuning-fig2.svg)

*Figure 2: Fine-tuning in the space of model parameters. The model parameters the method can return lie in the ball $\overline{\mathcal{B}}_{B}(\widehat{{\mathbf w}}^{(\mathrm{pre})})$ (dashed circle) rather than anywhere in the hypothesis space; a few gradient descent steps on a small training set cannot carry them further. Reaching the same region from an arbitrary initialization (dotted path) would take far more labeled data points*

The radius $B$ is what makes a small training set enough. Consider a linear model, whose hypotheses are $h^{({\mathbf w})}({\mathbf x}) = {\mathbf w}^{\top}
{\mathbf x}$. Let the feature vectors have norm at most $R$, and let the loss be $\rho$-Lipschitz in the prediction and bounded by $c$ on the ball, as the hinge loss is. Let the data points be realizations of independent and identically distributed RVs. Then, with probability at least $1 - \delta$, every ${\mathbf w} \in \overline{\mathcal{B}}_{B}(\widehat{{\mathbf w}}^{(\mathrm{pre})})$ satisfies

$$
\bar{L} \big( h^{({\mathbf w})} \big)
\leq f({\mathbf w})
+ \frac{2 \rho B R}{\sqrt{m}}
+ c \sqrt{\frac{2 \ln(2/\delta)}{m}}
\text{.}
$$

This is the bound of Shalev-Shwartz and Ben-David (2014, Thm. 26.12) for model parameters of norm at most $B$, applied to the shifted model parameters ${\mathbf w} - \widehat{{\mathbf w}}^{(\mathrm{pre})}$. The shift adds the fixed number $(\widehat{{\mathbf w}}^{(\mathrm{pre})})^{\top} {\mathbf x}$ to the prediction for each data point, which the contraction step of the proof allows, as it admits a separate $\rho$-Lipschitz function per data point (Shalev-Shwartz and Ben-David, 2014, Lemma 26.9). The second term follows from the Rademacher complexity of the ball, which is at most $B R / \sqrt{m}$ (Shalev-Shwartz and Ben-David, 2014, Lemma 26.10).

The generalization gap between risk and training error $f({\mathbf w})$ therefore grows in proportion to the radius $B$ and not with the number $d$ of model parameters. A gap of at most $\varepsilon$ is guaranteed by $m = \big\lceil \big(2 \rho B R + c
\sqrt{2 \ln(2/\delta)}\big)^{2} / \varepsilon^{2} \big\rceil$ data points, a number that falls with $B$. The constraint restricts the model the method searches, which is what regularization achieves by pruning a hypothesis space. The difference is that the pruned set is centered at the pretrained model parameters instead of at an arbitrary initialization.

Gradient descent with few steps and a small step size, started at the pretrained model parameters, implements this constraint without enforcing it explicitly. Each step moves the model parameters by $\eta$ times the norm of the gradient. After $T$ steps with step size $\eta$, started at ${\mathbf w}^{(0)} = \widehat{{\mathbf w}}^{(\mathrm{pre})}$,

$$
\Vert {{\mathbf w}^{(T)} - \widehat{{\mathbf w}}^{(\mathrm{pre})}} \Vert
\leq \eta \sum_{t=0}^{T-1}
\Vert {\nabla f({\mathbf w}^{(t)})} \Vert
\leq \eta T G
\text{,}
$$

where $G$ bounds the norm of the gradient along the iterates. The iterate therefore lies in the ball of radius $B = \eta T G$ around $\widehat{{\mathbf w}}^{(\mathrm{pre})}$. Limiting the number of iterations is thus a form of regularization: it limits the effective hypothesis space that is searched during fine-tuning, with the step size and the iteration count in the role of the radius.

Fine-tuning is useful when the task-specific dataset is small. A hospital has a few hundred annotated scans and needs a tumor detector. Training a deep net from an arbitrary initialization on that many labeled data points usually ends in overfitting; starting from model parameters already learned from millions of unrelated photographs achieves a lower validation error (Goodfellow et al., 2016, Sect. 15.2; Yosinski et al., 2014). LLMs are adapted the same way: one pretrained on unlabeled text is fine-tuned on each task it is wanted for (Devlin et al., 2019). A common variant freezes some layers and adapts only the remainder (Yosinski et al., 2014); parameter-efficient fine-tuning restricts the update to a low-dimensional set of new model parameters while leaving $\widehat{{\mathbf w}}^{(\mathrm{pre})}$ untouched (Hu et al., 2022).

The experiment pythondemos/finetuning.py measures the effect on synthetic data for linear regression with $m = 15$ data points and $d = 30$ model parameters. Gradient descent started at the pretrained model parameters with step size $\eta = 0.02$ falls to a validation error below twice the noise level after $T =
16$ steps. It has then moved the model parameters a distance $0.32$, inside the ball of radius $\eta T G = 0.72$. The same iteration from an arbitrary initialization drives the training error to zero yet ends with a validation error more than four times larger, having moved a distance $2.6$. Fig. 3 sketches the same effect on a schematic task-specific dataset of four data points.

![Figure 3 of the entry finetuning](https://dictionaryofml.org/terms/fig/finetuning-fig3.svg)

*Figure 3: Fine-tuning on a task-specific dataset of four data points. The pretrained hypothesis (dashed line) is already close to the task; a few gradient descent steps on the four data points of the task training set (dots) move it to the solid curve. From an arbitrary initialization, the four data points alone would have to determine the whole curve*

**Synonyms:** fine tuning.

**See also:** [pretraining](https://dictionaryofml.org/terms/pretraining.html), [transfer learning](https://dictionaryofml.org/terms/transferlearning.html), foundation model, [self-supervised learning](https://dictionaryofml.org/terms/selfsupervisedlearning.html), [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), [gradient descent](https://dictionaryofml.org/terms/gd.html), [regularization](https://dictionaryofml.org/terms/regularization.html), Rademacher complexity, [overfitting](https://dictionaryofml.org/terms/overfitting.html), generalization gap, [validation error](https://dictionaryofml.org/terms/valerr.html).

## References

- Bishop and Bishop (2024). Deep Learning: Foundations and Concepts. *Springer Nature*. <https://doi.org/10.1007/978-3-031-45468-4>
- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Goodfellow et al. (2016). Deep Learning. *MIT Press*.
- Yosinski et al. (2014). How Transferable Are Features in Deep Neural Networks?. *Adv. Neural Inf. Process. Syst. (NIPS)*.
- Devlin et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. *Proc. 2019 Conf. North American Chapter Assoc. Computational Linguistics: Human Lang. Technologies, Volume 1 (Long and Short Papers)*. <https://doi.org/10.18653/v1/N19-1423>
- Hu et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. *Int. Conf. Learn. Represent. (ICLR)*.

## Cite this entry

```bibtex
@misc{dictml_finetuning,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {fine-tuning},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/finetuning.html}
}
```


---

# Gaussian random variable (Gaussian RV)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/gaussrv.html) · [typeset PDF](https://dictionaryofml.org/terms/gaussrv.pdf) · [LaTeX](https://dictionaryofml.org/terms/gaussrv.tex) · [Python demo](https://dictionaryofml.org/terms/gaussrv.py) · [notebook](https://dictionaryofml.org/terms/gaussrv.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A Gaussian random variable (Gaussian RV) is a real-valued random variable whose probability density function is proportional to the exponential of a negative quadratic. The probability density function is determined by two numbers, the mean $\mu$ and the variance $\sigma^{2}$ of the random variable; the probability distribution with these two parameters is denoted by $\mathcal{N}\left(\mu,\sigma^{2}\right)$. The standard Gaussian random variable is the special case $\mu = 0$ and $\sigma^{2} = 1$; a general Gaussian random variable is obtained from it by scaling and shifting. A Gaussian random vector is an affine transformation of independent standard Gaussian RVs. Gaussian RVs arise as limits of averages of many independent RVs through the central limit theorem. Among all random vectors with a given covariance matrix, the Gaussian random vector maximizes the differential entropy.

## Definition

The measurement error of a sensor, such as a thermometer, is usually modeled as a Gaussian random variable. A real-valued random variable $z$ is a standard Gaussian random variable if its probability density function is (Bertsekas and Tsitsiklis, 2008, Sect. 3.3; Gray, 2009; Papoulis and Pillai, 2002)

$$
p^{(z)}\left(\eta\right) = \frac{1}{\sqrt{2\pi}} \exp\,(-\eta^2/2) \text{.}
$$

Given a standard Gaussian random variable $z$, a general Gaussian random variable $x$ with mean $\mu$ and variance $\sigma^2$ can be constructed via $x := \sigma z + \mu$, so that the probability distribution of $x$ is the normal distribution $\mathcal{N}\left(\mu,\sigma^{2}\right)$, denoted by $x \sim \mathcal{N}\left(\mu,\sigma^{2}\right)$ (Bertsekas and Tsitsiklis, 2008, Sect. 3.3). The probability density function of $x$ is, for every $\eta \in \mathbb{R}$,

$$
p^{(x)}\left(\eta\right) = \frac{1}{\sqrt{2\pi\sigma^{2}}}
\exp\Big( -\frac{(\eta-\mu)^{2}}{2\sigma^{2}} \Big) \text{.}
$$

The standard Gaussian random variable is the special case $\mu = 0$ and $\sigma^{2} = 1$.

The two parameters of $\mathcal{N}\left(\mu,\sigma^{2}\right)$ act separately on the probability density function, as shown in Fig. 1: the parameter $\mu$ is the mean of $x$, and the parameter $\sigma$ is its standard deviation, the square root of the variance $\sigma^{2}$. Adding $\mu$ to the scaled random variable $\sigma z$ shifts the probability density function along the horizontal axis by $\mu$, without changing its shape. For $\sigma>0$, multiplying $z$ by $\sigma$ stretches the probability density function horizontally by a factor of $\sigma$ and lowers its peak by the same factor, so the area under it remains $1$.

![Figure 1 of the entry gaussrv](https://dictionaryofml.org/terms/fig/gaussrv-fig1.svg)

*Figure 1: The solid curve is the probability density function of a standard Gaussian random variable $z$. The dashed curve is the probability density function of the Gaussian random variable $x = \sigma z + \mu$ with $\mu = 2$ and $\sigma = 1.5$. The shift parameter $\mu$ moves the mean from $0$ to $\mu$, while the scale parameter $\sigma$ changes the standard deviation from $1$ to $\sigma$ and the variance from $1$ to $\sigma^{2}$. A vertical ruler marks the mean of each curve, at $\eta = 0$ and at $\eta = \mu$, and each horizontal double arrow runs from the mean of its own curve to one standard deviation away, $1$ for the solid curve and $\sigma$ for the dashed one. The gray arrow from peak to peak marks the shift and the scaling that carry the one probability density function to the other*

Two ways to generate a Gaussian random variable from other sources of randomness are in common use. For exact generation, let $U$ be a random variable uniformly distributed on the open interval $(0,1) \subset \mathbb{R}$ and let $\Phi$ be the cumulative distribution function of a standard Gaussian random variable. Then $\mu + \sigma\Phi^{-1}(U) \sim \mathcal{N}\left(\mu,\sigma^{2}\right)$, the same probability distribution as $x$ but obtained from a different source of randomness.

For approximate generation, let $m$ be a positive integer and let $v^{(1)},\,\ldots,\,v^{(m)}$ be independent and identically distributed RVs, not necessarily Gaussian, with mean $0$ and finite, nonzero variance $\tau^{2}$. Then the normalized sum

$$
\frac{1}{\tau\sqrt{m}}\sum_{i=1}^{m}v^{(i)}
$$

converges in distribution to a standard Gaussian random variable as $m$ increases (Bertsekas and Tsitsiklis, 2008, Sect. 5.4; Ross, 2014, Thm. 3.1, p. 391). Thus, the central limit theorem makes a Gaussian approximation appropriate for averages of many independent contributions, which is why the measurement error of a sensor is usually modeled as Gaussian, although a finite sum need not be Gaussian (Ross, 2014). Fig. 2 shows both halves of that statement for coin flips, each taking the values $+1$ and $-1$ with probability $1/2$. The probability mass function of the normalized sum of $m = 8$ of them already follows the shape of the standard Gaussian probability density function, and the sum still takes only $9$ values. Convergence in distribution is convergence of the cdfs, so the largest gap between the cumulative distribution function of the normalized sum and that of a standard Gaussian random variable measures how far the approximation still is. It is $0.137$ for $m = 8$ and $0.050$ for $m = 64$.

![Figure 2 of the entry gaussrv](https://dictionaryofml.org/terms/fig/gaussrv-fig2.svg)

*Figure 2: Here, $s$ is the sum of $m = 8$ independent and identically distributed RVs, each taking the values $+1$ and $-1$ with probability $1/2$ (coin flips). The stems carry the probability mass function of the normalized sum $s/\sqrt{m}$, which takes $9$ values spaced $2/\sqrt{m}$ apart, so the normalized sum is not Gaussian. The dashed curve is the probability a standard Gaussian random variable puts on the cell of that width around each value, its probability density function there times $2/\sqrt{m}$. Data generated by pythondemos/gaussrv.py*

A Gaussian random vector ${\mathbf x} \in \mathbb{R}^{d}$ with covariance matrix ${\mathbf C}$ and mean ${\boldsymbol \mu}$ can be constructed as (Gray, 2009; Lapidoth, 2017; Papoulis and Pillai, 2002)

$$
{\mathbf x} := {\mathbf A} {\mathbf z} + {\boldsymbol \mu} \text{.}
$$

Here, ${\mathbf z} := \big( z_{1}, \,\ldots, \,z_{d} \big)^{\top}$ is a vector of independent and identically distributed standard Gaussian RVs. The matrix ${\mathbf A} \in \mathbb{R}^{d \times d}$ is a matrix square root of ${\mathbf C}$, i.e., any matrix with ${\mathbf A} {\mathbf A}^{\top} = {\mathbf C}$ (a Cholesky factor, for instance). The probability distribution of a Gaussian random vector is referred to as the multivariate normal distribution, denoted by $\mathcal{N}\left({\boldsymbol \mu},{\mathbf C}\right)$. A Gaussian random vector need not have independent components; its components are independent exactly when its covariance matrix is diagonal.

A Gaussian random vector ${\mathbf x}=\big(x_{1},\,\ldots,\,x_{d}\big)^{\top}$ can be interpreted as a stochastic process indexed by the finite set $\{1,\,\ldots,\,d\}$. More generally, a Gaussian process is a stochastic process over an index set $\mathcal{I}$ whose restriction to every finite subset of $\mathcal{I}$ is a Gaussian random vector (Rasmussen and Williams, 2006).

A standard Gaussian random variable $z$ obeys the Gaussian tail bound $\mathbb{P}\left(|z| \geq t\right) \leq 2 \exp(-t^{2}/2)$ for every positive number $t$ (Wainwright, 2019, Example 2.1, Eq. (2.9)). That bound also controls functions of independent and identically distributed RVs. Consider a standard Gaussian random vector ${\mathbf z} \in \mathbb{R}^{d}$ and a function $f :
\mathbb{R}^{d} \to \mathbb{R}$ that is Lipschitz with constant $L$, i.e., $\big| f({\mathbf z}) - f({\mathbf z}') \big| \leq
L \Vert {{\mathbf z} - {\mathbf z}'} \Vert$ for every pair of vectors ${\mathbf z}, {\mathbf z}' \in \mathbb{R}^{d}$. The deviation of $f({\mathbf z})$ from its expectation is then a sub-Gaussian random variable with parameter at most $L$. For every positive number $t$,

$$
\mathbb{P}\left(\big| f({\mathbf z}) - \mathbb{E}\big\{ f({\mathbf z}) \big\} \big| \geq t\right)
\leq 2 \exp\Big( \frac{-t^{2}}{2L^{2}} \Big)
\text{.}
$$

The number $d$ of arguments does not appear in this concentration inequality (Wainwright, 2019, Thm. 2.26): the bound is the same for two arguments and for a thousand. It constrains the tails of $f({\mathbf z})$ and does not state that $f({\mathbf z})$ has a normal distribution. The Euclidean norm is an example with $L = 1$: the length $\Vert {{\mathbf z}} \Vert$ has expectation at most $\sqrt{d}$ and deviates from it by more than $t$ with probability at most $2\exp(-t^{2}/2)$, however large $d$ is.

Among all random vectors with a given covariance matrix ${\mathbf C}$, the Gaussian random vector ${\mathbf x} \sim \mathcal{N}\left({\boldsymbol \mu},{\mathbf C}\right)$ maximizes the differential entropy (Cover and Thomas, 2006, Thm. 8.6.5). This property is one reason why additive Gaussian measurement noise is a widely used probabilistic model in the statistical analysis of machine learning methods: when only the variance of the noise is known, a Gaussian is the least restrictive assumption.

**See also:** multivariate normal distribution, Gaussian process, probabilistic model, central limit theorem, differential entropy, concentration inequality, sub-Gaussian random variable.

## References

- Bertsekas and Tsitsiklis (2008). Introduction to Probability. *Athena Scientific*.
- Gray (2009). Probability, Random Processes, and Ergodic Properties. *Springer Science+Business Media*. <https://doi.org/10.1007/978-1-4419-1090-5>
- Papoulis and Pillai (2002). Probability, Random Variables, and Stochastic Processes. *McGraw-Hill Higher Education*.
- Ross (2014). A First Course in Probability. *Pearson Education*.
- Lapidoth (2017). A Foundation in Digital Communication. *Cambridge Univ. Press*.
- Rasmussen and Williams (2006). Gaussian Processes for Machine Learning. *MIT Press*. <https://doi.org/10.7551/mitpress/3206.001.0001>
- Wainwright (2019). High-Dimensional Statistics: A Non-Asymptotic Viewpoint. *Cambridge Univ. Press*.
- Cover and Thomas (2006). Elements of Information Theory. *Wiley*. <https://doi.org/10.1002/047174882X>

## Cite this entry

```bibtex
@misc{dictml_gaussrv,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {Gaussian random variable (Gaussian RV)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/gaussrv.html}
}
```


---

# gradient descent (GD)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/gd.html) · [typeset PDF](https://dictionaryofml.org/terms/gd.pdf) · [LaTeX](https://dictionaryofml.org/terms/gd.tex) · [Python demo](https://dictionaryofml.org/terms/gd.py) · [notebook](https://dictionaryofml.org/terms/gd.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Gradient descent (GD) is an iterative algorithm for minimizing a differentiable function $f: \mathbb{R}^{d} \rightarrow \mathbb{R}$. Each iteration updates the current estimate by a step along the negative gradient, scaled by a step size $\eta$. Such a gradient step can be interpreted as the application of an operator that is parameterized by the step size and the underlying function. GD and its variants are widely used to train parametric models in deep learning.

## Definition

When training a deep net for image recognition, the average loss incurred by the network on a training set of labeled images is a function $f({\mathbf w})$ of the vector ${\mathbf w} \in \mathbb{R}^{d}$ of its model parameters, and training means minimizing that function. Many machine learning applications amount to such an optimization problem: an objective function $f: \mathbb{R}^{d} \rightarrow \mathbb{R}$ measures the quality of the model parameters ${\mathbf w}$, which are the optimization variable. Minimizing $f$ on the training set is a proxy: the goal is a small loss on data points outside it (see generalization).

In many cases, the objective function is differentiable: at every point ${\mathbf w}$ there exists a gradient $\nabla f({\mathbf w}) \in \mathbb{R}^{d}$, the vector of partial derivatives of $f$ at ${\mathbf w}$, which determines a local linear approximation of $f$ (see differentiable).

GD uses this local linear approximation to improve the current choice of the model parameters iteratively. Starting from an initialization ${\mathbf w}^{(0)}$, it generates a sequence of estimates ${\mathbf w}^{(0)}, {\mathbf w}^{(1)}, {\mathbf w}^{(2)}, \ldots$ that ideally converge to a minimum of $f$. At each iteration $t$, GD refines the current estimate ${\mathbf w}^{(t)}$ by stepping in the direction of steepest descent of the local linear approximation to $f$ at ${\mathbf w}^{(t)}$. This direction is the negative gradient $\nabla f({\mathbf w}^{(t)})$, giving the update

\begin{equation}
\tag{1}
{\mathbf w}^{(t+1)} = {\mathbf w}^{(t)}
- \eta \nabla f({\mathbf w}^{(t)}) \text{,}
\end{equation}

where $\eta > 0$ is a step size; the update is a gradient step with step size $\eta$. For a sufficiently small $\eta$, each step decreases the function value, $f({\mathbf w}^{(t+1)}) \le f({\mathbf w}^{(t)})$, with strict decrease whenever $\nabla f({\mathbf w}^{(t)}) \neq \mathbf{0}$ (Boyd and Vandenberghe, 2004, Sect. 9.3). GD and its variants are the standard solvers for model training in deep learning (Goodfellow et al., 2016). Fig. 1 illustrates a single GD step. Algorithm 1 states the method in full.

![Algorithm 1 of the entry gd](https://dictionaryofml.org/terms/fig/gd-alg1.svg)

*Algorithm 1: Gradient descent*

![Figure 1 of the entry gd](https://dictionaryofml.org/terms/fig/gd-fig1.svg)

*Figure 1: A single gradient step Eq. (1) toward the minimizer $\widehat{{\mathbf w}}$ of $f({\mathbf w})$. The curve depicts the function $f$. The straight line is its local linear approximation at the current estimate ${\mathbf w}^{(t)}$. A change $\Delta {\mathbf w}$ of the estimate ${\mathbf w}^{(t)}$ changes this approximation by $\big(\Delta {\mathbf w}\big)^{\top} \nabla f({\mathbf w}^{(t)})$. Stepping against the gradient by $-\eta \nabla f({\mathbf w}^{(t)})$ yields the next estimate ${\mathbf w}^{(t+1)}$*

With a constant step size $\eta$, held fixed across iterations, a single step is described by the GD step operator $\mathcal{F}^{(\eta)}({\mathbf w}) = {\mathbf w} - \eta \nabla f({\mathbf w})$, so GD is the fixed-point iteration ${\mathbf w}^{(t+1)} =
\mathcal{F}^{(\eta)}({\mathbf w}^{(t)})$. Its fixed points, where $\mathcal{F}^{(\eta)}({\mathbf w}) = {\mathbf w}$, are exactly the stationary points $\nabla f({\mathbf w}) = \mathbf{0}$. For convex $f$, these are the minima according to the zero-gradient condition (Boyd and Vandenberghe, 2004, Sect. 4.2.3).

The convergence of the fixed-point iteration ${\mathbf w}^{(t+1)} = \mathcal{F}^{(\eta)}({\mathbf w}^{(t)})$ is governed by the properties of the GD step operator $\mathcal{F}^{(\eta)}$. Assume $f$ is convex and its gradient is $L$-Lipschitz continuous, i.e.,

$$
\Vert {\nabla f({\mathbf w}) - \nabla f({\mathbf w}')} \Vert \le
L\,\Vert {{\mathbf w} - {\mathbf w}'} \Vert
\quad \text{for all } {\mathbf w}, {\mathbf w}' \in \mathbb{R}^{d} \text{.}
$$

For $0 < \eta \le 1/L$, the operator $\mathcal{F}^{(\eta)}$ is then a non-expansive operator with respect to the Euclidean norm $\Vert {\cdot} \Vert$: it does not increase distances between any two points ${\mathbf w}, {\mathbf w}'$,

$$
\Vert {\mathcal{F}^{(\eta)}({\mathbf w}) - \mathcal{F}^{(\eta)}({\mathbf w}')} \Vert
\le \Vert {{\mathbf w} - {\mathbf w}'} \Vert \text{.}
$$

Non-expansiveness alone does not guarantee a unique fixed point (Bauschke and Combettes, 2011, Ch. 4).

Assume, moreover, that $f$ is $\mu$-strongly convex, i.e., the function $f({\mathbf w}) - (\mu/2) \Vert {{\mathbf w}} \Vert^{2}$ is still convex, so the curvature of $f$ is at least $\mu > 0$ in every direction. Then $\mathcal{F}^{(\eta)}$ is a contractive operator with respect to the same norm,

$$
\Vert {\mathcal{F}^{(\eta)}({\mathbf w}) - \mathcal{F}^{(\eta)}({\mathbf w}')} \Vert \le
(1 - \mu\eta)\,\Vert {{\mathbf w} - {\mathbf w}'} \Vert \text{.}
$$

In this case, Banach's fixed-point theorem gives a unique fixed point $\widehat{{\mathbf w}}$ (the minimizer of $f$) to which GD converges at a geometric rate (Boyd and Vandenberghe, 2004, Sect. 9.3; Bubeck, 2015, Thm. 3.10). For linear regression, these properties are explicit: the gradient of the average squared error loss is an affine function of ${\mathbf w}$, and the contraction factor of $\mathcal{F}^{(\eta)}$ is governed by the eigenvalues of the matrix ${\mathbf X}^{\top}{\mathbf X}$. Here, ${\mathbf X}$ denotes the feature matrix, whose rows are the feature vectors of the data points in the training set (see linear regression).

How fast GD converges depends on the curvature of $f$, as quantified by the constants $L$ and $\mu$ above. Let $\widehat{{\mathbf w}}$ be a minimizer and $f^{\star} = f(\widehat{{\mathbf w}})$. For convex $f$ with $L$-Lipschitz gradient and the constant step size $\eta = 1/L$ (Beck, 2017, Thm. 10.21),

$$
f\big({\mathbf w}^{(T)}\big) - f^{\star} \le
\frac{L\,\Vert {{\mathbf w}^{(0)} - \widehat{{\mathbf w}}} \Vert^{2}}{2\,T}
\text{,}
$$

so the suboptimality after $T$ steps falls in proportion to $1/T$: halving the suboptimality requires doubling the number of iterations. If $f$ is in addition $\mu$-strongly convex, the bound improves to the geometric rate

$$
f\big({\mathbf w}^{(T)}\big) - f^{\star} \le
\left(1 - \tfrac{\mu}{L}\right)^{T}
\big(f({\mathbf w}^{(0)}) - f^{\star}\big) \text{,}
$$

consistent with the contractive operator factor $1 - \mu\eta$ above (Boyd and Vandenberghe, 2004, Sect. 9.3; Bubeck, 2015, Thm. 3.10). A geometric rate means that the suboptimality shrinks by at least the constant factor $1 - \mu/L$ in every single iteration. Thus, halving the suboptimality now costs a fixed number of iterations, however small the current suboptimality already is. Momentum methods, discussed below, improve both bounds.

The iteration also has a continuous-time counterpart. Consider shrinking the step size $\eta$ in the GD step Eq. (1) toward zero. For $\eta \to 0$, the iterate ${\mathbf w}^{(t)}$ converges to ${\mathbf w}(t \eta)$, the value at time $t \eta$ of the continuous-time trajectory ${\mathbf w}(\tau)$ that satisfies the ordinary differential equation $\dot{{\mathbf w}}(\tau) = -\nabla f({\mathbf w}(\tau))$, known as gradient flow. In particular, GD is the explicit Euler discretization of this gradient flow with step $\eta$ (Helmke and Moore, 1994). Replacing the time derivative $\dot{{\mathbf w}}(\tau)$ by the difference quotient $\big({\mathbf w}^{(t+1)} - {\mathbf w}^{(t)}\big)/\eta$ recovers the update Eq. (1). Fig. 2 illustrates how, for a small step size, the GD iterates closely follow the gradient flow trajectory.

![Figure 2 of the entry gd](https://dictionaryofml.org/terms/fig/gd-fig2.svg)

*Figure 2: GD as the explicit Euler discretization of gradient flow, for the quadratic $f({\mathbf w}) = \big(w_{1}^{2} + 5\,w_{2}^{2}\big)/2$ with initialization ${\mathbf w}^{(0)} = (2, 1.8)^{\top}$. Gray ellipses are level sets of $f$; the solid curve is the exact gradient flow trajectory. For the small step size $\eta = 0.02$, the GD iterates (dashed, circular marks) follow the trajectory closely. For the large step size $\eta = 0.3$, the iterates (dotted, square marks) oscillate across the narrow direction of the level sets. Data generated by pythondemos/gd.py*

Returning to the discrete iteration, in two common settings that motivate variants of GD, the assumptions behind these guarantees fail or the gradient becomes too costly. First, when $f$ is non-smooth, its gradient may fail to exist and GD is replaced by subgradient descent, which steps along a subgradient in place of the gradient. A concrete case is the least absolute shrinkage and selection operator objective, whose penalty term $\alpha \left\lVert {\mathbf w} \right\rVert_{1}$ is non-smooth. Second, in empirical risk minimization-based methods, $f$ is an average of $m$ loss functions, one for each data point, so evaluating the full gradient sums $m$ gradients and its cost grows in proportion to the training set size $m$. GD is thus replaced by stochastic gradient descent, which uses a gradient estimate from a random subset of the training set at each step.

Momentum methods provide another generalization of plain GD: each update combines gradients from several past steps rather than the current one alone. The name stems from a physical analogy: the iterates trace a particle whose momentum accumulates the force exerted by the negative gradient of the objective function. One prototypical example of a momentum method is Polyak's heavy-ball method. It adds a fraction $\beta \in [0, 1)$ of the previous increment,

$$
{\mathbf w}^{(t+1)} = {\mathbf w}^{(t)}
- \eta \nabla f({\mathbf w}^{(t)})
+ \beta\big({\mathbf w}^{(t)} - {\mathbf w}^{(t-1)}\big)
\text{, for } t=1,\ldots.
$$

For a strongly convex quadratic $f$, this iteration converges faster than plain GD (Polyak, 1987, Sect. 3.2.1). Fig. 3 compares plain GD with the heavy-ball method on a strongly convex quadratic function.

![Figure 3 of the entry gd](https://dictionaryofml.org/terms/fig/gd-fig3.svg)

*Figure 3: Suboptimality $f({\mathbf w}^{(t)}) - f^{\star}$ of plain GD (solid, circular marks) and heavy-ball momentum (dashed, square marks) on a strongly convex quadratic, shown on a logarithmic scale. Both sequences decay geometrically, which appears as a straight line on this scale. The momentum method shrinks the suboptimality by a smaller factor per iteration and therefore reaches any prescribed accuracy in fewer iterations. Data generated by pythondemos/gd.py*

A related momentum algorithm is Nesterov's accelerated variant of GD, which improves the $1/T$ bound above for convex $f$ with $L$-Lipschitz gradient to

$$
f\big({\mathbf w}^{(T)}\big) - f^{\star} \le
\frac{2\,L\,\Vert {{\mathbf w}^{(0)} - \widehat{{\mathbf w}}} \Vert^{2}}
{(T+1)^{2}} \text{.}
$$

In contrast to the bound for plain gradient descent, which falls in proportion to $1/T$, this bound falls in proportion to $1/T^{2}$ (Beck and Teboulle, 2009, Thm. 4.4; Bubeck, 2015, Thm. 3.19). For $\mu$-strongly convex $f$, acceleration improves the contraction factor from $1 - \mu/L$ to $1 - \sqrt{\mu/L}$ (Nesterov, 2004, Sect. 2.2; Bubeck, 2015, Thm. 3.18).

**See also:** [gradient](https://dictionaryofml.org/terms/gradient.html), step size, [stochastic gradient descent](https://dictionaryofml.org/terms/stochGD.html), subgradient descent, subgradient, gradient flow, [convex](https://dictionaryofml.org/terms/convex.html), strongly convex, fixed point.

## References

- Boyd and Vandenberghe (2004). Convex Optimization. *Cambridge Univ. Press*. <https://doi.org/10.1017/CBO9780511804441>
- Goodfellow et al. (2016). Deep Learning. *MIT Press*.
- Bauschke and Combettes (2011). Convex Analysis and Monotone Operator Theory in Hilbert Spaces. *Springer Science+Business Media*. <https://doi.org/10.1007/978-1-4419-9467-7>
- Bubeck (2015). Convex Optimization: Algorithms and Complexity. *Found. Trends Mach. Learn.*. <https://doi.org/10.1561/2200000050>
- Beck (2017). First-Order Methods in Optimization. *SIAM-Society for Industrial and Applied Mathematics*.
- Helmke and Moore (1994). Optimization and Dynamical Systems. *Springer*. <https://doi.org/10.1007/978-1-4471-3467-1>
- Polyak (1987). Introduction to optimization. *Optimization Software, Inc., Publications Division, New York*.
- Beck and Teboulle (2009). A fast iterative shrinkage-thresholding algorithm for linear inverse problems. *SIAM J. Imaging Sci.*. <https://doi.org/10.1137/080716542>
- Nesterov (2004). Introductory Lectures on Convex Optimization: A Basic Course. *Kluwer Academic*. <https://doi.org/10.1007/978-1-4419-8853-9>

## Cite this entry

```bibtex
@misc{dictml_gd,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {gradient descent (GD)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/gd.html}
}
```


---

# generalization

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-07. [HTML](https://dictionaryofml.org/terms/generalization.html) · [typeset PDF](https://dictionaryofml.org/terms/generalization.pdf) · [LaTeX](https://dictionaryofml.org/terms/generalization.tex) · [Python demo](https://dictionaryofml.org/terms/generalization.py) · [notebook](https://dictionaryofml.org/terms/generalization.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Generalization is the ability of a machine learning method to make accurate predictions on data points that have not been used during training. Many machine learning methods implement training via empirical risk minimization, i.e., learning a hypothesis $\widehat{h}$ that minimizes the empirical risk on the training set $\mathcal{D}^{(\rm train)}$. The learned $\widehat{h}$ is then used to compute predictions on new data points outside $\mathcal{D}^{(\rm train)}$. A probabilistic model for data generation quantifies the prediction errors beyond $\mathcal{D}^{(\rm train)}$ via the risk of $\widehat{h}$, i.e., the expected loss incurred on a new randomly chosen data point. The difference between the risk and the empirical risk is the generalization gap. The stability of the machine learning method bounds the expected generalization gap without reference to the size of the hypothesis space. A complementary deterministic view characterizes generalization via the robustness of $\widehat{h}$: its prediction changes by at most a bounded multiple of a perturbation of the features of a data point.

## Definition

A weather forecaster trained on past data is useful only if its predictions are accurate on data points that did not belong to the training set. Generalization refers to the ability of a learned hypothesis $\widehat{h}$ to make similarly accurate predictions on data points that have not been used during training. It is the central goal of machine learning: learning patterns in the features and labels of data points that extend beyond the training set.

The canonical learning technique is empirical risk minimization: the machine learning system obtains $\widehat{h} \in \mathcal{H}$ by minimizing the empirical risk on the training set $\mathcal{D}^{(\rm train)}$. A low empirical risk on $\mathcal{D}^{(\rm train)}$ does not guarantee accurate predictions on data points outside $\mathcal{D}^{(\rm train)}$. The same question arises for online learning and Bayesian inference, where the training set is replaced by the data points processed so far or by a prior.

Studying generalization mathematically requires formalizing the notion of data points outside the training set. A widely used approach is to assume a probabilistic model for the generation of data points, such as the independent and identically distributed assumption. Here, data points are interpreted as independent RVs with an identical, fixed but unknown, probability distribution $\mathbb{P}$. Fig. 1 sketches this setup: the shaded ellipse depicts the region of typical data points under $\mathbb{P}$, and the two large filled circles are the data points of the training set that empirical risk minimization fits with the learned hypothesis $\widehat{h}$, and the arrows are small perturbations of them. The risk of $\widehat{h}$ is the expected loss:

$$
\bar{L} \big( \widehat{h} \big) = \mathbb{E}
\left\{ L\left({\mathbf z},\widehat{h} \right) \right\}.
$$

The difference $\bar{L} \big( \widehat{h} \big) -
\widehat{L}\big(\widehat{h}|\mathcal{D}^{(\rm train)}\big)$ is the generalization gap.

![Figure 1 of the entry generalization](https://dictionaryofml.org/terms/fig/generalization-fig1.svg)

*Figure 1: Two data points ${\mathbf z}^{(1)}, {\mathbf z}^{(2)}$ used as a training set to learn a hypothesis $\widehat{h}$ via empirical risk minimization. The shaded ellipse marks the region of typical data points under an assumed probability distribution $\mathbb{P}$*

For any fixed hypothesis $h \in \mathcal{H}$, $\bar{L} \big( h \big)$ is a deterministic number that depends only on $h$ and $\mathbb{P}$. In contrast, the empirical risk $\widehat{L}\big(h|\mathcal{D}^{(\rm train)}\big)$ is a random variable that depends on the training set $\mathcal{D}^{(\rm train)}$ of size $m$, which, in turn, is a realization of an independent and identically distributed sequence of RVs ${\mathbf z}^{(1)}, \ldots, {\mathbf z}^{(m)} \sim
\mathbb{P}$. Bounding the generalization gap therefore means controlling the probability of the (undesirable) event

$$
\mathcal{A}^{(h)} :=
\left\{ \mathcal{D}^{(\rm train)} :
\big| \widehat{L}\big(h|\mathcal{D}^{(\rm train)}\big) -
\bar{L} \big( h \big) \big| > \varepsilon \right\},
$$

where $\varepsilon > 0$ is a user-specified tolerance for the generalization gap. Each $h \in \mathcal{H}$ defines a different event $\mathcal{A}^{(h)}$ (see Fig. 2). Since the learned hypothesis $\widehat{h}$ depends on $\mathcal{D}^{(\rm train)}$, a bound on the probability of $\mathcal{A}^{(h)}$ for one fixed $h$ does not apply to it. It suffices that $\mathcal{A}^{(h)}$ has low probability uniformly over $\mathcal{H}$: outside the union of these events, every $h \in
\mathcal{H}$, and so also $\widehat{h}$, has a generalization gap of at most $\varepsilon$ (Shalev-Shwartz and Ben-David, 2014, Lemma 4.2; Vapnik, 2000). Fig. 2 draws these events for a feature space and a label space that are both the real line, as subsets of the sample space of all training sets of size $m$.

![Figure 2 of the entry generalization](https://dictionaryofml.org/terms/fig/generalization-fig2.svg)

*Figure 2: Consider a machine learning application with feature space $\mathcal{X}=\mathbb{R}$ and label space $\mathcal{Y}=\mathbb{R}$. A training set of size $m$ is a point in the sample space $(\mathbb{R} \times \mathbb{R})^{m}$ of all possible training sets of size $m$ (the outer region). Each hypothesis $h \in \mathcal{H}$ defines an event $\mathcal{A}^{(h)} \subseteq (\mathbb{R} \times \mathbb{R})^{m}$ (an inner region) collecting those training sets on which $h$ fails to generalize. For the machine learning method to generalize, it suffices that the union $\bigcup_{h \in \mathcal{H}} \mathcal{A}^{(h)}$ has low probability; a low probability of a single $\mathcal{A}^{(h)}$ does not suffice, since the learned hypothesis depends on the training set*

Controlling every event $\mathcal{A}^{(h)}$ uniformly over $\mathcal{H}$ is not the only probabilistic route. If the machine learning method is stable, i.e., replacing a single data point of the training set changes the loss of the learned hypothesis only slightly on average, then the expected generalization gap equals that average change, regardless of the size of the hypothesis space (see stability; Shalev-Shwartz and Ben-David, 2014, Thm. 13.2).

Probability theory is not the only route to generalization. A complementary deterministic approach uses robustness: a good hypothesis $\widehat{h}$ should change its prediction $\widehat{h}({\mathbf x})$ by at most a bounded multiple of the perturbation of the feature vector ${\mathbf x}$ of a data point ${\mathbf z}$, i.e., it should be Lipschitz. For example, an object detection system trained on smartphone photos should still detect the object if a single pixel is changed. Trained deep nets have been shown to lack this property (Su et al., 2019). The system should also deliver the same result if the object in the image is slightly translated or deformed (Mallat, 2016, Sect. 3). The arrows in Fig. 1 indicate small perturbations of each data point in the training set of a machine learning method.

**See also:** generalization gap, risk, empirical risk, [independent and identically distributed assumption](https://dictionaryofml.org/terms/iidasspt.html), [training set](https://dictionaryofml.org/terms/trainset.html), [overfitting](https://dictionaryofml.org/terms/overfitting.html), [regularization](https://dictionaryofml.org/terms/regularization.html), stability.

## References

- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Vapnik (2000). The Nature of Statistical Learning Theory. *Springer*. <https://doi.org/10.1007/978-1-4757-3264-1>
- Su et al. (2019). One Pixel Attack for Fooling Deep Neural Networks. *IEEE Trans. Evol. Comput.*. <https://doi.org/10.1109/TEVC.2019.2890858>
- Mallat (2016). Understanding deep convolutional networks. *Philos. Trans. Roy. Soc. A*. <https://doi.org/10.1098/rsta.2015.0203>

## Cite this entry

```bibtex
@incollection{dictml_generalization,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {generalization},
  booktitle = {Dictionary of Applied Machine Learning},
  publisher = {Springer Nature Singapore},
  year = {2026},
  doi = {10.1007/978-981-92-2012-0_7-1},
  isbn = {978-981-92-2012-0},
  url = {https://doi.org/10.1007/978-981-92-2012-0_7-1}
}
```


---

# Gaussian mixture model (GMM)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-10. [HTML](https://dictionaryofml.org/terms/gmm.html) · [typeset PDF](https://dictionaryofml.org/terms/gmm.pdf) · [LaTeX](https://dictionaryofml.org/terms/gmm.tex) · [Python demo](https://dictionaryofml.org/terms/gmm.py) · [notebook](https://dictionaryofml.org/terms/gmm.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A Gaussian mixture model (GMM) is a probabilistic model for the generation of data points with numeric feature vectors. Each data point is generated by first drawing a latent cluster index and then drawing the feature vector from the multivariate normal distribution of that cluster. The resulting marginal distribution is a weighted sum of multivariate normal distributions. The model parameters are chosen to maximize the likelihood function of a dataset. The resulting maximum likelihood optimization problem is an instance of empirical risk minimization using a specific choice for the loss function. This optimization problem is non-convex and typically solved using the expectation–maximization algorithm. This algorithm consists of a fixed-point iteration that never increases the empirical risk.

## Definition

A photograph of a mountain is cut into small square patches, and each patch is described by two numbers: how green and how blue it is on average (Fig. 1). Vegetation, mountain and haze, and bright sky form three clusters of such patches, which are data points characterized by a feature vector ${\mathbf x} \in \mathbb{R}^{d}$. A GMM is a probabilistic model for the generation of such data points (Bishop, 2006; Murphy, 2012).

A GMM assumes that each data point is generated by first drawing a latent cluster index $I \in \{1,\,\ldots,\,k\}$ according to cluster probabilities

$$
\mathbb{P}\left(I=c\right) = p_{c},
\qquad
\sum_{c=1}^{k} p_{c}=1\text{.}
$$

Given the choice $I=c$, the feature vector ${\mathbf x}$ is drawn from a multivariate normal distribution $\mathbb{P}^{(c)}= \mathcal{N}\left({\boldsymbol \mu}^{(c)},\mathbf{C}^{(c)}\right)$, with mean ${\boldsymbol \mu}^{(c)}$ and covariance matrix $\mathbf{C}^{(c)}$. The resulting marginal distribution of ${\mathbf x}$ is therefore a weighted sum of multivariate normal distributions (Fig. 1), i.e.,

$$
\mathbb{P}
=\sum_{c=1}^{k}
p_{c} \mathcal{N}\left({\boldsymbol \mu}^{(c)},\mathbf{C}^{(c)}\right)\text{.}
$$

![Figure 1 of the entry gmm](https://dictionaryofml.org/terms/fig/gmm-fig1.svg)

*Figure 1: Top: the photograph of the Oetscher massif with the patch grid drawn on it; each patch is one data point. Bottom: the $972$ patches, each described by its average greenness and its average blueness, both on a scale from $0$ to $1$. The three labeled curves, vegetation (solid curve), mountain and haze (dashed curve), and bright sky (dotted curve), are contours of the multivariate normal distributions of a GMM, so each covariance matrix shows as an ellipse, elongated because greenness and blueness are correlated within each cluster. Data generated by pythondemos/gmm.py*

A GMM is parameterized by the cluster-specific probability $p_{c} = \mathbb{P}\left(I=c\right)$, mean ${\boldsymbol \mu}^{(c)}$, and covariance matrix $\mathbf{C}^{(c)}$ for $c=1,\,\ldots,\,k$. These model parameters can be learned from a dataset $\mathcal{D} = \{{\mathbf x}^{(1)}, \ldots,
{\mathbf x}^{(m)}\}$ by maximum likelihood. The resulting optimization problem has no closed-form solution, and the standard method for solving it approximately is the expectation–maximization algorithm (Dempster et al., 1977), which alternates between two steps. The E-step computes, for every data point ${\mathbf x}^{(r)}$ and every cluster index $c$, the posterior distribution under the current model parameters,

\begin{equation}
\tag{1}
u_{c}^{(r)} :=
\mathbb{P}\left(I = c \mid {\mathbf x}^{(r)}\right) =
\frac{p_{c}\,
\mathcal{N}\left({\boldsymbol \mu}^{(c)},\mathbf{C}^{(c)}\right)({\mathbf x}^{(r)})}
{\sum_{c'=1}^{k} p_{c'}\,
\mathcal{N}\left({\boldsymbol \mu}^{(c')},\mathbf{C}^{(c')}\right)({\mathbf x}^{(r)})}\text{,}
\end{equation}

and the M-step re-fits the model parameters as averages weighted by these posterior distributions,

\begin{equation}
\tag{2}
\begin{aligned}
p_{c} &:= \frac{1}{m}
\sum_{r=1}^{m} u_{c}^{(r)}, \qquad
{\boldsymbol \mu}^{(c)} :=
\frac{\sum_{r=1}^{m} u_{c}^{(r)}\, {\mathbf x}^{(r)}}
{\sum_{r=1}^{m} u_{c}^{(r)}}, \\
\mathbf{C}^{(c)} &:=
\frac{\sum_{r=1}^{m} u_{c}^{(r)}\,
({\mathbf x}^{(r)} - {\boldsymbol \mu}^{(c)})
({\mathbf x}^{(r)} - {\boldsymbol \mu}^{(c)})^{\top}}
{\sum_{r=1}^{m} u_{c}^{(r)}}
\quad \text{for } c = 1, \ldots, k\text{.}
\end{aligned}
\end{equation}

The M-step formulas Eq. (2) are not a closed-form solution of the maximum likelihood optimization problem, because the posterior distributions on their right-hand side depend on the model parameters through the E-step Eq. (1) (Bishop, 2006, Sect. 9.2.2). The two steps are repeated until the model parameters stop changing (Bishop, 2006, Sect. 9.2.2).

The maximum likelihood optimization problem for the model parameters of a GMM is an instance of empirical risk minimization with a particular choice for the loss function. Assume that every covariance matrix $\mathbf{C}^{(c)}$ is invertible, so that the GMM has a probability density function on $\mathbb{R}^{d}$. The loss incurred by a feature vector ${\mathbf x}$ is the negative logarithm of this probability density function at ${\mathbf x}$,

$$
L\left({\mathbf x},\mathbb{P} \right) = - \ln
\sum_{c=1}^{k} p_{c}
\mathcal{N}\left({\boldsymbol \mu}^{(c)},\mathbf{C}^{(c)}\right)
({\mathbf x}) \text{.}
$$

The empirical risk of a GMM is then obtained as the average loss over the entire dataset. Minimizing it maximizes the likelihood function, since the logarithm is increasing and the sign is flipped. Fitting a GMM is therefore empirical risk minimization over the hypothesis space of mixtures of $k$ multivariate normal distributions.

The loss function is a non-convex function of the model parameters (Murphy, 2012, Sect. 11.3.2). The expectation–maximization algorithm might therefore converge to a local minimum of the empirical risk (Bishop, 2006, Sect. 9.2.2), so it is typically run several times with different initializations, keeping the run with the smallest empirical risk (Murphy, 2012, Sect. 11.3.2). A common construction of the initial model parameters runs k-means first and starts from its cluster centroids (Bishop, 2006, Sect. 9.2.2).

Whatever its initialization, the expectation–maximization algorithm is, like k-means, a fixed-point iteration. Collect the model parameters of a GMM in one vector ${\mathbf w}$. One iteration, the E-step Eq. (1) followed by the M-step Eq. (2), applies an operator $\mathcal{F}$ to the current model parameters, so the method is the fixed-point iteration ${\mathbf w}^{(t+1)} = \mathcal{F}\big({\mathbf w}^{(t)}\big)$ for $t = 0,1,\ldots$. The fixed points of $\mathcal{F}$ are model parameters at which the gradient of the empirical risk vanishes, i.e., the stationary points of the empirical risk (see zero-gradient condition) (Wu, 1983, Sect. 2.1). Convergence follows from monotone descent rather than from a contractive operator property: applying $\mathcal{F}$ never increases the empirical risk (Dempster et al., 1977, Thm. 1), so the sequence of empirical risk values is non-increasing. It converges if the empirical risk is bounded below. This bound is not automatic for a GMM: a component whose mean coincides with one data point and whose variance shrinks to zero drives the empirical risk to $-\infty$ (Bishop, 2006, Sect. 9.2.1). The bound holds once the eigenvalues of every covariance matrix $\mathbf{C}^{(c)}$ are at least some $\delta > 0$, since then the probability density function of the GMM nowhere exceeds $(2\pi\delta)^{-d/2}$. Under the conditions of Wu (1983, Thm. 2), which imply this bound, the sequence of empirical risk values converges and every limit point of the iterates ${\mathbf w}^{(t)}$ is a fixed point. On the patches of Fig. 1, the empirical risk never increased in any of the $300$ iterations of expectation–maximization.

Trained by expectation–maximization on the $972$ patches of Fig. 1, a three-component GMM separates vegetation, mountain and haze, and bright sky. Every patch it assigns to the bluest component lies in the upper half of the photograph. The learned model parameters define a soft clustering: the posterior distribution $\mathbb{P}\left(I = c \mid {\mathbf x}\right)$ is the probability that a data point belongs to cluster $c$, and it quantifies the degree of belonging to cluster $c$. In contrast, hard clustering methods assign each data point to exactly one cluster. Fig. 2 shows these degrees of belonging on the photograph itself.

![Figure 2 of the entry gmm](https://dictionaryofml.org/terms/fig/gmm-fig2.svg)

*Figure 2: (a) The photograph from Fig. 1 at patch resolution, each patch one data point. The same photograph three times, each patch dimmed by its posterior distribution for one cluster: vegetation (b), mountain and haze (c), and bright sky (d). A patch that belongs to one cluster alone is bright in one copy and black in the other two, while a shared patch is dim in several. Because the three probabilities of a patch sum to one, the three copies add back up to (a). Data generated by pythondemos/gmm.py*

The expectation–maximization algorithm for learning the model parameters of a GMM reduces to Lloyd's algorithm for k-means under additional constraints. Fix every cluster probability at $1/k$ and every covariance matrix at $\sigma^{2} \mathbf{I}$, a multiple of the identity matrix with one common $\sigma^{2}$. The posterior distribution then depends on ${\mathbf x}$ only through its Euclidean distances to the means, and as $\sigma^{2}$ shrinks towards $0$ it concentrates on the closest one. In the limit ($\sigma^{2} \to 0$), the E-step Eq. (1) becomes an assignment of every data point to its nearest mean, and the M-step Eq. (2) re-centers the means on the data points assigned to them (Bishop, 2006, Sect. 9.3.2, Eq. (9.42)). That is Lloyd's algorithm for k-means.

Fig. 3 shows the effect of the common $\sigma^{2}$ on the degree of belonging to the vegetation cluster. With $\sigma^{2} = 10^{-4}$, every patch obtains a degree of belonging of $0$ or $1$, and the grading coincides with the k-means assignment on more than $99\%$ of the patches. With $\sigma^{2} = 2 \cdot 10^{-2}$, $63\%$ of the patches obtain a degree of belonging between $0.2$ and $0.8$. On the patches of Fig. 1, with $\sigma^{2} = 1$ the three means collapse onto the average of all patches and every patch obtains the same degree of belonging $1/3$. Learning the covariance matrices and the cluster probabilities instead of fixing them lets the grading adapt to the data, where k-means assigns each data point to one cluster.

![Figure 3 of the entry gmm](https://dictionaryofml.org/terms/fig/gmm-fig3.svg)

*Figure 3: The photograph at patch resolution, each patch dimmed by its degree of belonging to the vegetation cluster under the constrained GMM with cluster probabilities $1/k$ and covariance matrices $\sigma^{2} \mathbf{I}$. (a) The k-means assignment: a patch is bright if it is assigned to the vegetation cluster and black otherwise. (b) $\sigma^{2} = 10^{-4}$: the grading is again $0$ or $1$ and coincides with (a). (c) $\sigma^{2} = 2 \cdot 10^{-2}$: the meadow is graded highest, the foliage and the horizon in between, the sky lowest. (d) $\sigma^{2} = 1$: every patch is graded $1/3$. Data generated by pythondemos/gmm.py*

**See also:** probabilistic model, multivariate normal distribution, [clustering](https://dictionaryofml.org/terms/clustering.html), [expectation–maximization](https://dictionaryofml.org/terms/em.html), fixed-point iteration, maximum likelihood, soft clustering, degree of belonging, hard clustering, [$k$-means](https://dictionaryofml.org/terms/kmeans.html), Lloyd's algorithm.

## References

- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Murphy (2012). Machine Learning: A Probabilistic Perspective. *MIT Press*.
- Dempster et al. (1977). Maximum likelihood from incomplete data via the EM algorithm. *J. Roy. Statist. Soc.: Ser. B (Methodological)*. <https://doi.org/10.1111/j.2517-6161.1977.tb01600.x>
- Wu (1983). On the convergence properties of the EM algorithm. *Ann. Statist.*. <https://doi.org/10.1214/aos/1176346060>

## Cite this entry

```bibtex
@misc{dictml_gmm,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {Gaussian mixture model (GMM)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/gmm.html}
}
```


---

# gradient

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/gradient.html) · [typeset PDF](https://dictionaryofml.org/terms/gradient.pdf) · [LaTeX](https://dictionaryofml.org/terms/gradient.tex) · [Python demo](https://dictionaryofml.org/terms/gradient.py) · [notebook](https://dictionaryofml.org/terms/gradient.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

The gradient of a real-valued function is a vector that determines the local linear approximation of the function. The entries of the gradient are the partial derivatives of the function. Their existence does not by itself make the function differentiable: the vector they form need not give a local linear approximation. For a convex function, the existence of all partial derivatives at a point already makes it differentiable there. Geometrically, a nonzero gradient is orthogonal to the level sets and points in the direction of steepest ascent. In machine learning, gradients of the empirical risk minimization objective function define the update of gradient descent methods and are computed for deep networks by backpropagation.

## Definition

A linear model for the maximum temperature at Krems an der Donau has two model parameters, and changing either of them changes the average squared error loss on the training set. Training a machine learning model adjusts its model parameters ${\mathbf w}$ to reduce a loss $f({\mathbf w})$, which requires knowing how $f$ changes as ${\mathbf w}$ moves. The gradient of a real-valued function $f: \mathbb{R}^{d} \rightarrow \mathbb{R}: {\mathbf w} \mapsto f({\mathbf w})$ answers this by determining a local linear approximation of $f$. Formally, the gradient of $f$ at a point ${\mathbf w}' \in \mathbb{R}^{d}$ is a vector ${\mathbf g} \in \mathbb{R}^{d}$ such that

\begin{equation}\tag{1}
\lim_{{\mathbf w} \rightarrow {\mathbf w}'}
\frac{f({\mathbf w}) - \big( f({\mathbf w}') + {\mathbf g}^{\top} ({\mathbf w} - {\mathbf w}') \big)}{\left\lVert {\mathbf w} - {\mathbf w}' \right\rVert_{2}}
= 0 \text{.}
\end{equation}

If such a vector exists, it is unique, the function $f$ is differentiable at ${\mathbf w}'$, and the vector is denoted by $\nabla f({\mathbf w}')$ or $\nabla f({\mathbf w}) \big|_{{\mathbf w}'}$ (Rudin, 1976, Def. 9.11 and Thm. 9.12). The function $f({\mathbf w}') + \big(\nabla f({\mathbf w}')\big)^{\top} ({\mathbf w} - {\mathbf w}')$ is the local linear approximation of $f$ at ${\mathbf w}'$ (see Fig. 1).

![Figure 1 of the entry gradient](https://dictionaryofml.org/terms/fig/gradient-fig1.svg)

*Figure 1: A differentiable function $f: \mathbb{R} \rightarrow \mathbb{R}$ (thick curve) and its local linear approximation at a point ${\mathbf w}'$ (thin straight line). The dashed slope triangle shows the displacement ${\mathbf w} - {\mathbf w}'$ (horizontal leg) and the resulting change $\big(\nabla f({\mathbf w}')\big)^{\top} ({\mathbf w} - {\mathbf w}')$ (vertical leg) of the local linear approximation*

The entries of the gradient are the partial derivatives of $f$, $\nabla f({\mathbf w}) = \big( {\partial f}/{\partial w_{1}}, \ldots, {\partial f}/{\partial w_{d}} \big)^{\top}$ (Rudin, 1976, Thm. 9.17); the partial derivative ${\partial f}/{\partial w_{j}}$ is the rate of change of $f$ when only the $j$-th entry of ${\mathbf w}$ varies. The existence of all partial derivatives of $f$ at a point does not guarantee that the gradient exists there (Rudin, 1976, Def. 9.16 and Thm. 9.21). Additional assumptions on $f$ restore the implication; convexity is one sufficient condition. If a convex function $f: \mathbb{R}^{d} \rightarrow \mathbb{R}$ has partial derivatives $\partial f / \partial w_{j}$, for $j = 1, \ldots, d$, at a point ${\mathbf w}'$, then $f$ is differentiable at ${\mathbf w}'$, and the gradient $\nabla f({\mathbf w}')$ is the vector of these partial derivatives (Rockafellar, 1970, Thm. 25.2).

At every point where $f$ is differentiable and $\nabla f \neq \mathbf{0}$, the gradient is orthogonal to the level set of $f$ through that point, the set $\{{\mathbf w} : f({\mathbf w}) = c\}$ of points where $f$ takes a common value $c$. It also points in the direction of steepest ascent of $f$. The negative gradient $-\nabla f({\mathbf w}')$ points in the direction of steepest descent, and gradient descent repeatedly steps along it (see Fig. 2; Boyd and Vandenberghe, 2004, Sect. 9.4.1). At a local minimum of a differentiable function $f: \mathbb{R}^{d} \rightarrow \mathbb{R}$, there cannot be any direction of descent and consequently the gradient must vanish (Nesterov, 2004, Thm. 1.2.1) (see zero-gradient condition).

![Figure 2 of the entry gradient](https://dictionaryofml.org/terms/fig/gradient-fig2.svg)

*Figure 2: Level sets $f({\mathbf w}) = c$ (the curves along which $f$ equals a constant $c$) of a differentiable function $f$ of two variables $w_{1}, w_{2}$ (axes, lower left), with a minimum at $\widehat{{\mathbf w}}$. The gradient $\nabla f({\mathbf w}')$ (solid arrow) is orthogonal to the level set through ${\mathbf w}'$ (the curve through ${\mathbf w}'$) and points in the direction of steepest ascent. The negative gradient $-\nabla f({\mathbf w}')$ (dashed arrow) points in the direction of steepest descent, toward smaller values of $f$. Data generated by pythondemos/gradient.py*

Gradients are used in the training of machine learning models. They determine the update of model parameters that minimizes the incurred loss on a training set $\mathcal{D}^{(\rm train)} = \big\{ \big({\mathbf x}^{(r)}, y^{(r)}\big) \big\}_{r=1}^{m}$, which is the goal of empirical risk minimization. For example, linear regression minimizes the objective function defined by

$$
f({\mathbf w}) := \frac{1}{m}
\sum_{r=1}^{m}
\big( y^{(r)} -
{\mathbf w}^{\top} {\mathbf x}^{(r)} \big)^{2} \text{.}
$$

This function is convex and differentiable, with gradient

$$
\nabla f({\mathbf w}) = - \frac{2}{m}
\sum_{r=1}^{m}
\big( y^{(r)} -
{\mathbf w}^{\top} {\mathbf x}^{(r)} \big)
{\mathbf x}^{(r)} \text{.}
$$

For a deep artificial neural network whose activation functions, loss, and every other operation it is composed of are differentiable, the empirical risk minimization objective function is also differentiable, by the chain rule (Rudin, 1976, Thm. 9.15). The gradient of this objective function can be computed by backpropagation (Goodfellow et al., 2016, Sect. 6.5; Rumelhart et al., 1986).

The definition carries over to a real-valued function $f: \mathcal{H} \rightarrow \mathbb{R}$ on a real Hilbert space $\mathcal{H}$, a Hilbert space over $\mathbb{R}$. In the defining limit Eq. (1), the inner product $\langle {\mathbf g}, {\mathbf w} - {\mathbf w}' \rangle$ of $\mathcal{H}$ replaces the term ${\mathbf g}^{\top} ({\mathbf w} - {\mathbf w}')$. The norm of $\mathcal{H}$ replaces the Euclidean norm $\left\lVert {\mathbf w} - {\mathbf w}' \right\rVert_{2}$ (Bauschke and Combettes, 2011, Remark 2.44 and Def. 2.45).

**See also:** function, vector, differentiable, partial derivative, [gradient descent](https://dictionaryofml.org/terms/gd.html), zero-gradient condition, [convex](https://dictionaryofml.org/terms/convex.html), [Hilbert space](https://dictionaryofml.org/terms/hilbertspace.html).

## References

- Rudin (1976). Principles of Mathematical Analysis. *McGraw-Hill*.
- Rockafellar (1970). Convex Analysis. *Princeton Univ. Press*. <https://doi.org/10.1515/9781400873173>
- Boyd and Vandenberghe (2004). Convex Optimization. *Cambridge Univ. Press*. <https://doi.org/10.1017/CBO9780511804441>
- Nesterov (2004). Introductory Lectures on Convex Optimization: A Basic Course. *Kluwer Academic*. <https://doi.org/10.1007/978-1-4419-8853-9>
- Goodfellow et al. (2016). Deep Learning. *MIT Press*.
- Rumelhart et al. (1986). Learning Representations by Back-Propagating Errors. *Nature*. <https://doi.org/10.1038/323533a0>
- Bauschke and Combettes (2011). Convex Analysis and Monotone Operator Theory in Hilbert Spaces. *Springer Science+Business Media*. <https://doi.org/10.1007/978-1-4419-9467-7>

## Cite this entry

```bibtex
@misc{dictml_gradient,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {gradient},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/gradient.html}
}
```


---

# Hilbert space

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-11. [HTML](https://dictionaryofml.org/terms/hilbertspace.html) · [typeset PDF](https://dictionaryofml.org/terms/hilbertspace.pdf) · [LaTeX](https://dictionaryofml.org/terms/hilbertspace.tex) · [Python demo](https://dictionaryofml.org/terms/hilbertspace.py) · [notebook](https://dictionaryofml.org/terms/hilbertspace.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A Hilbert space is a complete inner product space: every Cauchy sequence of its elements has a limit in the space. Three examples in machine learning are: a finite-dimensional Euclidean space, a space of finite-variance RVs on a common probability space, and a reproducing kernel Hilbert space. Linear regression uses a Euclidean space as its feature space and to parameterize its hypothesis space. Optimal linear and nonlinear estimation of a random variable are orthogonal projections in the Hilbert space of finite-variance RVs. A kernel method uses a reproducing kernel Hilbert space as its transformed feature space and as its hypothesis space.

## Definition

Consider gradient descent for linear regression, or any iterative training method whose hypothesis space is a metric space: it produces the sequence $h^{(1)}, h^{(2)}, \ldots$ of hypotheses. For the sequence to converge to an optimal hypothesis $h^{\star}$, the hypotheses must form a Cauchy sequence (Rudin, 1976, Thm. 3.11, part (a)): for every distance $\epsilon > 0$, however small, there is an index $N$ such that the distance between $h^{(r)}$ and $h^{(r')}$ is below $\epsilon$ for all $r, r' \geq N$ (Rudin, 1976, Def. 3.8). Being a Cauchy sequence is not sufficient on its own. If the underlying metric space is complete, then every Cauchy sequence in that space converges to a point in the space. An example of a complete metric space is a Euclidean space $\mathbb{R}^{d}$ of finite dimension $d$.

A Hilbert space is a generalization of a Euclidean space to possibly infinite dimensions. In particular, a Hilbert space $(\mathcal{H}, \langle \cdot, \cdot \rangle)$ is an inner product space that is complete, i.e., in which every Cauchy sequence of its elements has a limit that again belongs to $\mathcal{H}$ (Rudin, 1987, Def. 4.4) (see Fig. 1).

![Figure 1 of the entry hilbertspace](https://dictionaryofml.org/terms/fig/hilbertspace-fig1.svg)

*Figure 1: Completeness. The elements ${\mathbf w}^{(1)}, {\mathbf w}^{(2)}, \ldots$ of a Cauchy sequence (dots) lie ever closer to each other, and their limit ${\mathbf w} = \lim_{t \rightarrow \infty} {\mathbf w}^{(t)}$ (cross) again belongs to the Hilbert space. Data generated by pythondemos/hilbertspace.py*

The elements of $\mathcal{H}$ are called vectors, whether they are arrays of numbers, RVs, or functions: what makes them vectors is that they can be added and scaled (see vector space). The inner product induces a norm $\left\lVert {\mathbf u} \right\rVert_{\mathcal{H}} := \sqrt{\langle {\mathbf u}, {\mathbf u} \rangle}$ and, in turn, a metric $d({\mathbf u},{\mathbf v}) := \left\lVert {\mathbf u} - {\mathbf v} \right\rVert_{\mathcal{H}}$ (Rudin, 1987, Sects. 4.2–4.3, Def. 4.4) (see inner product), so a Cauchy sequence of $\mathcal{H}$ is one whose vectors eventually lie within any prescribed norm distance of each other.

Three examples of Hilbert spaces are used throughout machine learning: a Euclidean space $\mathbb{R}^{d}$ as feature space, a space of RVs with finite variance on a common probability space, and a reproducing kernel Hilbert space of functions. The first of them carries the standard inner product $\langle {\mathbf u}, {\mathbf v} \rangle = {\mathbf u}^{\top} {\mathbf v}$, and its completeness follows from that of the real numbers (Rudin, 1976, Thm. 3.11). Linear regression uses the Euclidean space $\mathbb{R}^{d}$ in two roles. The feature vectors ${\mathbf x} \in \mathbb{R}^{d}$ of the data points it is applied to are vectors of this space, and so are the weights ${\mathbf w} \in \mathbb{R}^{d}$ that parameterize its hypothesis space $\big\{ h({\mathbf x}) = \langle {\mathbf w}, {\mathbf x} \rangle
\, : \, {\mathbf w} \in \mathbb{R}^{d} \big\}$. Gradient descent searching for those weights produces a sequence ${\mathbf w}^{(1)}, {\mathbf w}^{(2)}, \ldots$ of model parameters, the sequence depicted in Fig. 1. These are vectors in the second role, so the condition of the opening paragraph is met here: $\mathbb{R}^{d}$ is complete, and a Cauchy sequence of model parameters converges to model parameters.

The second example is the set of all RVs $x$ with finite variance, defined on a common probability space (Gray and Davisson, 2004, Sect. 5.8.1). Here, two RVs are identified whenever the expectation of their squared difference is zero, $\mathbb{E}\big\{ (x - x')^{2} \big\} = 0$, i.e., whenever $x = x'$ with probability one. With this identification, the expectation $\langle x, x' \rangle := \mathbb{E}\{ x x' \}$ is an inner product (Gray and Davisson, 2004, Sect. 5.8.1), and the resulting space is complete (Gray and Davisson, 2004, Lemma 5.1).

In the Hilbert space of finite-variance RVs on a common probability space, optimal linear estimation is an orthogonal projection (see Fig. 2). A linear estimator $\widehat{y}$ of a random variable $y$ from an observed random variable $x$ is an element of the subspace $\big\{ a x \, : \, a \in \mathbb{R} \big\}$ spanned by $x$. The corresponding estimation error $y - \widehat{y}$ is measured by the induced norm $\sqrt{\langle y - \widehat{y}, y - \widehat{y} \rangle}$. The smallest error is obtained by the linear estimator whose error is orthogonal to $x$ (Gray and Davisson, 2004, Thm. 4.9),

\begin{align}
0 & = \langle y - \widehat{y}, x \rangle \nonumber \\
& = \mathbb{E} \big\{ \big(y - \widehat{y} \big) x \big\} \text{.} \nonumber
\end{align}

When the RVs $y,x$ are zero-mean, this orthogonality means that the error $y - \widehat{y}$ is uncorrelated with $x$.

![Figure 2 of the entry hilbertspace](https://dictionaryofml.org/terms/fig/hilbertspace-fig2.svg)

*Figure 2: Optimal linear estimation as an orthogonal projection. The random variable $y$, the observed random variable $x$, and the estimator $\widehat{y}$ are vectors of the Hilbert space of finite-variance RVs. The estimators that are linear in $x$ form the subspace $\{ a x : a \in \mathbb{R} \}$ spanned by $x$ (the straight line through it), and the estimation error $y - \widehat{y}$ is smallest for the $\widehat{y}$ whose error is orthogonal to $x$, the right angle marked by the small square at $\widehat{y}$*

Optimal nonlinear estimation can also be represented as an orthogonal projection, but on a larger subspace. Projecting $y$ onto the RVs that are functions of $x$, rather than onto the multiples $a x$ alone, gives the conditional expectation $\mathbb{E}\{ y \mid x \}$: it has the smallest squared error among all estimators computed from $x$ (Gray and Davisson, 2004, Thm. 4.5).

The third example is a reproducing kernel Hilbert space. It is a Hilbert space $\mathcal{H}$ of functions $h: \mathcal{X} \to \mathbb{R}$ whose inner product reproduces point evaluations. In particular, each reproducing kernel Hilbert space is associated with a kernel $k\big(\cdot,\cdot\big)$ such that $k\big({\mathbf x},\cdot\big) \in \mathcal{H}$ for every ${\mathbf x} \in \mathcal{X}$ and

$$
h({\mathbf x})
= \langle h, k\big({\mathbf x},\cdot\big) \rangle
\quad \text{for every } h \in \mathcal{H}
\text{.}
$$

Thus, each vector $h \in \mathcal{H}$ is itself a hypothesis map $\mathcal{X} \to \mathbb{R}$, evaluated by taking an inner product with $k\big({\mathbf x},\cdot\big)$. This construction of a hypothesis map generalizes the linear model, which uses the standard inner product of the Euclidean space. The map ${\mathbf x} \mapsto k\big({\mathbf x},\cdot\big)$ is a feature transformation, carrying the feature vector of a data point into $\mathcal{H}$. A kernel method uses a reproducing kernel Hilbert space in two roles. It is the transformed feature space, in which a data point is represented by $k\big({\mathbf x},\cdot\big)$, and it is the hypothesis space from which a hypothesis is learned (Hastie et al., 2009, Sect. 12.3.3; Schölkopf and Smola, 2002). These are the two roles of $\mathbb{R}^{d}$ in linear regression, with one difference: there a vector parameterizes a hypothesis, here a vector is one.

The three examples differ in what their vectors are, not in their geometry. Every Hilbert space has an orthonormal basis (Rudin, 1987, Thm. 4.22), and any two Hilbert spaces whose orthonormal bases have the same number of elements are copies of one another: mapping one basis onto the other extends to a linear map that preserves inner products and, in turn, distances. Such a linear map is therefore an isometry (Rudin, 1987, Thms. 4.17 and 4.18, Sect. 4.19).

**See also:** [inner product](https://dictionaryofml.org/terms/innerproduct.html), vector space, [norm](https://dictionaryofml.org/terms/norm.html), Cauchy sequence, Euclidean space, reproducing kernel Hilbert space, orthogonal projection, isometry.

## References

- Rudin (1976). Principles of Mathematical Analysis. *McGraw-Hill*.
- Rudin (1987). Real and Complex Analysis. *McGraw-Hill*.
- Gray and Davisson (2004). An Introduction to Statistical Signal Processing. *Cambridge Univ. Press*. <https://ee.stanford.edu/~gray/sp.pdf>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Schölkopf and Smola (2002). Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. *MIT Press*. <https://doi.org/10.7551/mitpress/4175.001.0001>

## Cite this entry

```bibtex
@incollection{dictml_hilbertspace,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {Hilbert space},
  booktitle = {Dictionary of Applied Machine Learning},
  publisher = {Springer Nature Singapore},
  year = {2026},
  doi = {10.1007/978-981-92-2012-0_26-1},
  isbn = {978-981-92-2012-0},
  url = {https://doi.org/10.1007/978-981-92-2012-0_26-1}
}
```


---

# hyperparameter

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-10-29. [HTML](https://dictionaryofml.org/terms/hyperparameter.html) · [typeset PDF](https://dictionaryofml.org/terms/hyperparameter.pdf) · [LaTeX](https://dictionaryofml.org/terms/hyperparameter.tex) · [Python demo](https://dictionaryofml.org/terms/hyperparameter.py) · [notebook](https://dictionaryofml.org/terms/hyperparameter.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A hyperparameter is any quantity that specifies the components of a machine learning method. Examples are the maximum degree in polynomial regression, the regularization strength in ridge regression, and the number of data points used to construct a training set. Hyperparameters can also determine the behavior of the optimization method used to solve empirical risk minimization, that is, to train a model; one example is the learning rate in gradient descent. A hyperparameter selects one machine learning method out of several alternatives and can therefore be chosen by model selection techniques, such as those based on the validation error.

## Definition

Predicting the maximum temperature of a day at Krems an der Donau from its minimum temperature calls for decisions before any training happens: which maps the method may return, how many data points to collect, and how to search those maps. Fig. 1 shows twelve days as data points, each with the minimum temperature as its feature $x$ and the maximum temperature as its label $y$. Three straight lines depict hypotheses of the underlying hypothesis space, or model, of the machine learning method. A loss function is used to measure the prediction error incurred on a data point.

A hyperparameter is any quantity that specifies one of these components of a machine learning method. The maximum degree in polynomial regression specifies the model, the regularization strength in ridge regression affects the loss function (Bishop, 2006, Sect. 1.3), and the number of data points gathered specifies the training set.

![Figure 1 of the entry hyperparameter](https://dictionaryofml.org/terms/fig/hyperparameter-fig1.svg)

*Figure 1: Twelve days of 2020 at Krems an der Donau (filled circles), with the minimum temperature of a day as feature $x$ and its maximum temperature as label $y$. The three straight lines depict hypotheses of the model: the solid line is one hypothesis $h$, the dashed and dotted lines are two others; the dashed loop encircling them marks the hypothesis space $\mathcal{H}$. The vertical arrow marks the prediction error of a single data point under $h$, measured by the loss function $L\left((x,y),h \right)$. Data generated by pythondemos/hyperparameter.py*

The third decision of the opening, how to search the model, is specified by hyperparameters of the optimization method that solves empirical risk minimization. The learning rate of gradient descent, for example, bounds the distance between the iterate of gradient descent, minimizing the training error, and its starting point after a fixed number of gradient steps, and a larger learning rate raises that bound. Thus, the set of model parameters it can reach, and with them the effective hypothesis space, grows. Fig. 2 depicts this for five gradient steps on one training set, starting from the zero initialization ${\mathbf w}^{(0)} = (0,0)^{\top}$, at three values $\eta_{1} < \eta_{2} < \eta_{3}$ of the learning rate. The effective hypothesis spaces $\mathcal{H}^{(\eta_{1})} \subset
\mathcal{H}^{(\eta_{2})} \subset \mathcal{H}^{(\eta_{3})}$ are nested, so the learning rate indexes a family of models ordered by size, as the maximum degree does in polynomial regression.

![Figure 2 of the entry hyperparameter](https://dictionaryofml.org/terms/fig/hyperparameter-fig2.svg)

*Figure 2: Gradient descent on one training set of $30$ data points with two features, starting from the zero initialization ${\mathbf w}^{(0)} = (0,0)^{\top}$ (cross) and stopped after five gradient steps, for a small, a medium and a large learning rate, $\eta_{1} = 0.1/L$, $\eta_{2} = 0.4/L$ and $\eta_{3} = 1.4/L$, with $L$ the largest curvature of the training error. The dots are the iterates, drawn in the style of their learning rate: dashed for $\eta_{1}$, dash-dotted for $\eta_{2}$, solid for $\eta_{3}$. The ellipse $\mathcal{H}^{(\eta_{k})}$ passes through the iterate of learning rate $\eta_{k}$ farthest from ${\mathbf w}^{(0)}$ and has the shape of the level sets of the training error; it is the effective hypothesis space that learning rate allows and contains every gradient step taken at it. The three ellipses are nested, and each contains every iterate of every smaller learning rate. Data generated by pythondemos/hyperparameter.py*

Whichever component it specifies, a hyperparameter selects one machine learning method out of a family of alternatives, and that is what makes a single rule for choosing it possible.

The usefulness of a specific hyperparameter value can be assessed by model selection techniques such as validation (Hastie et al., 2009, Sect. 7.10). As with learning the model parameters by empirical risk minimization on a training set, a hyperparameter is chosen by minimizing the validation error, with the validation error in place of the training error (Bishop, 2006, Sect. 1.3). The search often runs over a grid or a randomly drawn set of candidate values, keeping the one with the smallest validation error on the validation set.

Fig. 3 shows this choice for the degree of a polynomial model learned by polynomial regression from $20$ data points. The training error falls with every increase of the degree and would select the largest one. The validation error on $100$ further data points is smallest at degree $3$ and rises steeply beyond it, so validation selects degree $3$.

![Figure 3 of the entry hyperparameter](https://dictionaryofml.org/terms/fig/hyperparameter-fig3.svg)

*Figure 3: Training error (solid line, filled circles) and validation error (dashed line, open squares) of a polynomial model against its degree, the hyperparameter. The training error never increases with the degree and would select degree $9$; the validation error is smallest at degree $3$, the value validation selects. Data generated by pythondemos/hyperparameter.py*

**See also:** [model](https://dictionaryofml.org/terms/model.html), validation, model parameter, [validation error](https://dictionaryofml.org/terms/valerr.html), [training error](https://dictionaryofml.org/terms/trainerr.html), [model selection](https://dictionaryofml.org/terms/modelsel.html), [learning rate](https://dictionaryofml.org/terms/learnrate.html), [gradient descent](https://dictionaryofml.org/terms/gd.html), polynomial regression.

## References

- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>

## Cite this entry

```bibtex
@misc{dictml_hyperparameter,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {hyperparameter},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/hyperparameter.html}
}
```


---

# hypothesis space

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/hypospace.html) · [typeset PDF](https://dictionaryofml.org/terms/hypospace.pdf) · [LaTeX](https://dictionaryofml.org/terms/hypospace.tex) · [Python demo](https://dictionaryofml.org/terms/hypospace.py) · [notebook](https://dictionaryofml.org/terms/hypospace.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A hypothesis space $\mathcal{H}$ is a set of hypothesis maps $h: \mathcal{X} \rightarrow \mathcal{Y}$ from a feature space into a label space. Every machine learning method uses an underlying hypothesis space, which is a subset of the set $\mathcal{Y}^{\mathcal{X}}$ of all possible maps from $\mathcal{X}$ into $\mathcal{Y}$. Available computational resources limit the size of $\mathcal{H}$, which in turn determines the method's computational cost and generalization gap. The hypothesis space typically carries geometric structure: real-valued hypotheses can be compared via the deviation in their predictions on a reference dataset, and a parametric model inherits a geometric structure from the underlying parameter space $\mathcal{W}$.

## Definition

A weather service predicting tomorrow's maximum daytime temperature from today's morning temperature must commit to a set of candidate prediction maps: for example, all linear functions of the morning temperature. This set of candidates is the hypothesis space of the method; more generally, every machine learning method commits to a specific hypothesis space $\mathcal{H}$, the set from which it learns a single hypothesis $\widehat{h}$. Formally, $\mathcal{H}$ is a subset of the set $\mathcal{Y}^{\mathcal{X}}$ of all possible maps from the feature space into the label space (see Fig. 1).

![Figure 1 of the entry hypospace](https://dictionaryofml.org/terms/fig/hypospace-fig1.svg)

*Figure 1: Hypothesis space $\mathcal{H}$ of a machine learning method as a restricted subset of the much larger set $\mathcal{Y}^{\mathcal{X}}$ of all possible maps from the feature space $\mathcal{X}$ into the label space $\mathcal{Y}$*

Linear regression and other linear methods use the linear model as $\mathcal{H}$, i.e., the set of all linear maps $\mathbb{R}^{d} \rightarrow \mathbb{R}$ (Shalev-Shwartz and Ben-David, 2014, Ch. 2). Another canonical example is the set of input–output maps realizable by an artificial neural network of fixed architecture, as the model parameters vary.

The same $\mathcal{H}$ can be used by different machine learning algorithms, such as empirical risk minimization on a fixed training set, online learning on a stream of data points, or Bayesian inference that returns a posterior distribution over $\mathcal{H}$ rather than a single hypothesis (Bishop, 2006, Sect. 1.2.3).

The choice of $\mathcal{H}$ is a central design decision for a machine learning method. From a machine learning engineering perspective, it is guided by two factors. The first is the available computational resources, such as memory, processing time, and communication bandwidth. The second factor is the number of available data points, which limits the maximum size of $\mathcal{H}$ that can be effectively trained without overfitting. In principle, $\mathcal{H}$ could be as large as $\mathcal{Y}^{\mathcal{X}}$ itself. The size of $\mathcal{H}$ allows the anticipation of generalization before any training is carried out. A small $\mathcal{H}$ typically incurs a small generalization gap, while a large $\mathcal{H}$ is more prone to overfitting (Shalev-Shwartz and Ben-David, 2014, Cor. 4.6, Thm. 6.8). For a finite $\mathcal{H}$, the effective size is its cardinality. For an infinite $\mathcal{H}$, an effective size can be quantified by the Vapnik–Chervonenkis dimension or the Rademacher complexity (Shalev-Shwartz and Ben-David, 2014, Thm. 6.8, Thm. 26.5).

The word *space* (rather than *set*) reflects the fact that $\mathcal{H}$ typically carries some geometric structure. In general, $\mathcal{H}$ is not a vector space: the sum of two hypotheses in $\mathcal{H}$ need not belong to $\mathcal{H}$. What is more, there are hypothesis spaces that are not equipped with an algebraic structure at all, i.e., there is no meaningful way to add two hypotheses in $\mathcal{H}$ or to multiply a hypothesis by a scalar.

For real-valued hypotheses, two $h, h' \in \mathcal{H}$ can be compared by their average squared prediction deviation on a reference dataset $\mathcal{D} = \{{\mathbf x}^{(1)}, \ldots, {\mathbf x}^{(m)}\}$,

$$
\frac{1}{m}
\sum_{r=1}^{m}
\big(h({\mathbf x}^{(r)})
- h'({\mathbf x}^{(r)})\big)^{2} \text{.}
$$

Fig. 2 illustrates this comparison: the two hypotheses are evaluated at the feature values of the data points in $\mathcal{D}$ and the squared differences in their predictions are averaged. The square root of this average, $d_{\mathcal{D}}(h,h')$, is a pseudo-metric on $\mathcal{H}$: it is the Euclidean norm of the difference of the two prediction vectors divided by $\sqrt{m}$, so it is symmetric and satisfies the triangle inequality, and it vanishes for two distinct hypotheses that agree on $\mathcal{D}$. This construction is not meaningful for hypotheses with a finite label space, as used in classification methods. Instead, two such hypotheses can be compared by their disagreement rate on $\mathcal{D}$.

![Figure 2 of the entry hypospace](https://dictionaryofml.org/terms/fig/hypospace-fig2.svg)

*Figure 2: Comparing two hypotheses $h$ (red) and $h'$ (blue) by their predictions at four scalar features $x^{(1)},\ldots,x^{(4)}$. The dashed segments mark the gap $h(x^{(r)}) - h'(x^{(r)})$ for each data point; the square root of the average of its squares is the pseudo-metric $d_{\mathcal{D}}(h,h')$ between $h$ and $h'$*

A parametric model $\mathcal{H} = \{h^{({\mathbf w})} : {\mathbf w} \in
\mathcal{W}\}$ inherits a geometric structure from the underlying parameter space $\mathcal{W} \subseteq \mathbb{R}^{d}$. A norm on $\mathbb{R}^{d}$ induces the candidate distance $d(h^{({\mathbf w})},h^{({\mathbf w}')})
:= \Vert {{\mathbf w} - {\mathbf w}'} \Vert$ on $\mathcal{H}$. When the parameterization ${\mathbf w} \mapsto h^{({\mathbf w})}$ is injective, distinct weight vectors give distinct hypotheses, and this candidate distance is a metric. The linear model $\mathcal{H} = \{h^{({\mathbf w})}({\mathbf x}) =
{\mathbf w}^{\top} {\mathbf x} : {\mathbf w} \in
\mathbb{R}^{d}\}$ is a canonical example of an injective parameterization: distinct weight vectors ${\mathbf w}$ produce distinct linear maps, so $\Vert {{\mathbf w} - {\mathbf w}'} \Vert$ is a metric on $\mathcal{H}$. When the parameterization is not injective, two distinct weight vectors ${\mathbf w} \neq {\mathbf w}'$ can represent the same hypothesis $h^{({\mathbf w})} =
h^{({\mathbf w}')}$. The distance $\Vert {{\mathbf w} - {\mathbf w}'} \Vert$ is then strictly positive even though the two hypotheses coincide, so it is only a pseudo-metric. Permuting the hidden units of an artificial neural network, for instance, changes ${\mathbf w}$ but leaves $h^{({\mathbf w})}$ unchanged (Bishop, 2006, Sect. 5.1.1).

**Synonyms:** hypothesis class, model.

**See also:** [hypothesis](https://dictionaryofml.org/terms/hypothesis.html), [model](https://dictionaryofml.org/terms/model.html), linear model, parametric model, parameter space, Vapnik–Chervonenkis dimension, Rademacher complexity, [empirical risk minimization](https://dictionaryofml.org/terms/erm.html).

## References

- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>

## Cite this entry

```bibtex
@incollection{dictml_hypospace,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {hypothesis space},
  booktitle = {Dictionary of Applied Machine Learning},
  publisher = {Springer Nature Singapore},
  year = {2026},
  doi = {10.1007/978-981-92-2012-0_6-1},
  isbn = {978-981-92-2012-0},
  url = {https://doi.org/10.1007/978-981-92-2012-0_6-1}
}
```


---

# hypothesis

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-07. [HTML](https://dictionaryofml.org/terms/hypothesis.html) · [typeset PDF](https://dictionaryofml.org/terms/hypothesis.pdf) · [LaTeX](https://dictionaryofml.org/terms/hypothesis.tex) · [Python demo](https://dictionaryofml.org/terms/hypothesis.py) · [notebook](https://dictionaryofml.org/terms/hypothesis.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A hypothesis is a map $h: \mathcal{X} \rightarrow \mathcal{Y}$ from the feature space $\mathcal{X}$ to the label space $\mathcal{Y}$. Given a data point with features ${\mathbf x}$, the hypothesis returns a prediction $h({\mathbf x})$ for the label $y$. A machine learning method selects a single learned hypothesis $\widehat{h}$ from a restricted hypothesis space $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$. This restriction reflects finite computational resources of a practical machine learning method. An empirical risk minimization-based method finds $\widehat{h}$ by a search that uses the structure of $\mathcal{H}$, e.g., gradient descent on a parametric model.

## Definition

Predicting tomorrow's maximum daytime temperature from today's morning temperature is a typical machine learning task. A hypothesis is the map that performs this prediction. Formally, a hypothesis is a map (also called a function) $h: \mathcal{X} \rightarrow \mathcal{Y}$ from the feature space $\mathcal{X}$ to the label space $\mathcal{Y}$ (Shalev-Shwartz and Ben-David, 2014, Sect. 2.1). Given a data point with features ${\mathbf x}$, the hypothesis $h$ returns an estimate $\widehat{y} = h({\mathbf x})$ of the true label $y$. The same structure, features in and prediction out, appears in a very different application: Fig. 1 shows a hypothesis that predicts a Freddie-likeness score from the samples of an audio recording, i.e., the measured amplitudes at successive time instants.

![Figure 1 of the entry hypothesis](https://dictionaryofml.org/terms/fig/hypothesis-fig1.svg)

*Figure 1: Hypothesis $h: \mathcal{X} \rightarrow \mathcal{Y}$ mapping the features ${\mathbf x} \in \mathcal{X}$ of a data point to a prediction $h({\mathbf x}) \in \mathcal{Y}$ of the label. Here, the machine learning application \urlhttps://freddiemeter.withyoutube.com/ uses the samples of an audio recording as features to predict how closely a person's singing resembles that of Freddie Mercury. The displayed value $h({\mathbf x})=0.82$ indicates an $82\%$ resemblance*

A machine learning method aims to find (or learn) a hypothesis $h$ such that $y \approx h({\mathbf x})$ for any data point with features ${\mathbf x}$ and label $y$. Finite computational resources require restricting the search to a subset of

$$
\mathcal{Y}^{\mathcal{X}} = \big\{ h: \mathcal{X} \rightarrow \mathcal{Y} \big\} \text{,}
$$

the set of all functions from $\mathcal{X}$ to $\mathcal{Y}$. This subset is the hypothesis space $\mathcal{H}$ (also called the model) underlying the method. The output of a machine learning algorithm acting on a training set is a single learned hypothesis $\widehat{h} \in \mathcal{H}$ obtained via empirical risk minimization, i.e., an element of $\mathcal{H}$ that minimizes the empirical risk on the training set (Shalev-Shwartz and Ben-David, 2014, Sect. 2.2). The search does not examine the hypotheses one by one. For a parametric model, whose hypotheses are indexed by a vector of weights, gradient descent moves through $\mathcal{H}$ by following the gradient of the empirical risk, which is possible only because $\mathcal{H}$ carries that structure.

Different hypothesis spaces admit different representations of their hypothesis maps. Polynomial regression (with a single scalar feature $x$) represents each $h \in \mathcal{H}$ as an algebraic expression $h(x) = \sum_{j=0}^{p} w_{j}
x^{j}$, where $p$ is the degree of the polynomial and the coefficients $w_{j}$ are the model parameters. A hypothesis space of computer programs represents $h$ as executable source code, e.g., a Python function with a prescribed signature. A decision tree or an artificial neural network represents $h$ as a signal-flow chart whose nodes apply simple operations (e.g., comparisons, weighted sums, nonlinear activations) and whose edges define the order in which these operations are composed. These representations differ in which functions they can express and in how efficiently the empirical risk minimization problem can be solved over the corresponding $\mathcal{H}$. Fig. 2 shows the three representations side by side.

![Figure 2 of the entry hypothesis](https://dictionaryofml.org/terms/fig/hypothesis-fig2.svg)

*Figure 2: Three representations of a hypothesis map. (a) As an algebraic expression, here a degree-2 polynomial in a scalar feature $x$. (b) As source code, here a Python function. (c) As a signal-flow chart, here a small decision tree that routes a data point to a prediction of $3$, $1$, or $-1$ based on the threshold comparisons on the feature $x$*

**Synonyms:** predictor.

**See also:** function, prediction, predictor, [hypothesis space](https://dictionaryofml.org/terms/hypospace.html), [data point](https://dictionaryofml.org/terms/datapoint.html), [label](https://dictionaryofml.org/terms/label.html), [empirical risk minimization](https://dictionaryofml.org/terms/erm.html).

## References

- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>

## Cite this entry

```bibtex
@incollection{dictml_hypothesis,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {hypothesis},
  booktitle = {Dictionary of Applied Machine Learning},
  publisher = {Springer Nature Singapore},
  year = {2026},
  doi = {10.1007/978-981-92-2012-0_5-1},
  isbn = {978-981-92-2012-0},
  url = {https://doi.org/10.1007/978-981-92-2012-0_5-1}
}
```


---

# independent and identically distributed (i.i.d.)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-07. [HTML](https://dictionaryofml.org/terms/iid.html) · [typeset PDF](https://dictionaryofml.org/terms/iid.pdf) · [LaTeX](https://dictionaryofml.org/terms/iid.tex) · [Python demo](https://dictionaryofml.org/terms/iid.py) · [notebook](https://dictionaryofml.org/terms/iid.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A collection of RVs is independent and identically distributed (i.i.d.) if all of them follow the same probability distribution and they are mutually independent. Two rolls of the same die are i.i.d.: both values have the same probability distribution, and the first says nothing about the second. The two requirements are separate, and each one can fail without the other. Mutual independence means that the probability of any combination of values is the product of one probability per random variable. This is stronger than independence of every pair. For jointly Gaussian RVs the two requirements hold exactly when all means are equal and the covariance matrix is a scaled identity matrix.

## Definition

Two rolls of the same die are independent and identically distributed: both values follow the same probability distribution, and the first says nothing about the second. A collection of RVs with these two properties is called i.i.d. Formally, consider RVs ${\mathbf z}^{(1)}, \ldots, {\mathbf z}^{(m)}$, all defined on one common probability space $(\Omega,\, \Sigma,\, \mathbb{P})$ and taking values in a measurable space $\mathcal{Z}$. They are i.i.d. if every ${\mathbf z}^{(r)}$ follows the same probability distribution and the RVs are mutually independent. The common probability space is what makes mutual independence well defined: one outcome has to fix all $m$ values at once (see random variable). The two requirements are separate, and each can fail without the other. Two sensors of different accuracy, read once each, deliver independent values that are not identically distributed. The temperature measured by one sensor at two consecutive times is identically distributed and dependent, since the second value almost repeats the first.

Mutual independence is reflected by a factorization of the probability distribution: the probability of any combination of their values is the product of one probability per random variable. For any measurable sets $\mathcal{A}_{1}, \ldots, \mathcal{A}_{m} \subseteq
\mathcal{Z}$,

\begin{equation}
\tag{1}
\mathbb{P}\left( {\mathbf z}^{(1)} \in \mathcal{A}_{1}, \ldots,
{\mathbf z}^{(m)} \in \mathcal{A}_{m}\right)
= \prod_{r=1}^{m}
\mathbb{P}\left( {\mathbf z}^{(r)} \in \mathcal{A}_{r}\right)
\text{.}
\end{equation}

The product rule Eq. (1) has to hold for the whole collection, not only for pairs. Let $b_{1}$ and $b_{2}$ be independent RVs, each taking the values $0$ and $1$ with probability $1/2$, and let $b_{3}$ be their sum modulo two. All three take the values $0$ and $1$ with probability $1/2$, so they are identically distributed, and any two of them are independent. The three together are not: $\mathbb{P}\left(b_{1} = 1, b_{2} = 1, b_{3} = 1\right) = 0$, because $1 + 1 = 0$ modulo two, while the product of the three single probabilities is $1/8$. Pairwise independence is therefore weaker than mutual independence, and these three RVs are identically distributed and pairwise independent without being i.i.d. (Gray and Davisson, 2004, Problem 4.31).

For jointly Gaussian RVs the two requirements take one compact form. Consider a random vector ${\mathbf x} = (x_{1}, \ldots, x_{d})^{\top}$ of jointly Gaussian RVs with mean vector ${\boldsymbol \mu}$ and covariance matrix ${\mathbf C}$. Its components are i.i.d. exactly when ${\boldsymbol \mu} = (\mu, \ldots, \mu)^{\top}$ and ${\mathbf C} = \sigma^{2} \mathbf{I}$ is a scaled identity matrix. Indeed, for ${\mathbf C} = \sigma^{2} \mathbf{I}$ the joint probability density function (Murphy, 2012, Eq. (4.1)) is a product of univariate Gaussian densities with means $\mu_{j}$ and the common variance $\sigma^{2}$, so the components are independent, and they are identically distributed exactly when the means agree. Conversely, i.i.d. components have equal means, equal variances, and zero covariances. Every component then has the same variance $\sigma^{2}$, whose square root $\sigma$ is the standard deviation along each axis. The off-diagonal zeros make the components uncorrelated. For jointly Gaussian RVs this already implies mutual independence (Anderson, 2003, Sect. 2.4), so the three RVs $b_{1}, b_{2}, b_{3}$ have no jointly Gaussian counterpart. The equal diagonal entries and the equal means make the components identically distributed. Fig. 1 shows realizations of such a random vector for $d = 2$.

![Figure 1 of the entry iid](https://dictionaryofml.org/terms/fig/iid-fig1.svg)

*Figure 1: Realizations of a pair $x_{1}, x_{2}$ of i.i.d. Gaussian RVs with mean $\mu$ and variance $\sigma^{2}$, each drawn as an open circle. The solid curve has radius $\sigma$ and is centered at $(\mu, \mu)$. Identical probability distributions make the spread the same along both axes, and mutual independence leaves the cloud untilted, so the covariance matrix is $\sigma^{2}$ times the identity matrix. Data generated by pythondemos/iid.py*

**See also:** [random variable](https://dictionaryofml.org/terms/rv.html), [independent and identically distributed assumption](https://dictionaryofml.org/terms/iidasspt.html), [probability distribution](https://dictionaryofml.org/terms/probdist.html), probability space, event, measurable, realization, [Gaussian random variable](https://dictionaryofml.org/terms/gaussrv.html), [covariance matrix](https://dictionaryofml.org/terms/covmtx.html), random vector, [variance](https://dictionaryofml.org/terms/variance.html).

## References

- Gray and Davisson (2004). An Introduction to Statistical Signal Processing. *Cambridge Univ. Press*. <https://ee.stanford.edu/~gray/sp.pdf>
- Murphy (2012). Machine Learning: A Probabilistic Perspective. *MIT Press*.
- Anderson (2003). An Introduction to Multivariate Statistical Analysis. *Wiley*.

## Cite this entry

```bibtex
@misc{dictml_iid,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {independent and identically distributed (i.i.d.)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/iid.html}
}
```


---

# independent and identically distributed assumption (i.i.d. assumption)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-11. [HTML](https://dictionaryofml.org/terms/iidasspt.html) · [typeset PDF](https://dictionaryofml.org/terms/iidasspt.pdf) · [LaTeX](https://dictionaryofml.org/terms/iidasspt.tex) · [Python demo](https://dictionaryofml.org/terms/iidasspt.py) · [notebook](https://dictionaryofml.org/terms/iidasspt.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

The independent and identically distributed assumption (i.i.d. assumption) is the probabilistic model that represents the generation of a sequence of data points by independent and identically distributed RVs. It consists of two separate requirements, joint independence and one common probability distribution. The assumption makes the average loss on a training set an unbiased estimate of the risk, which is what justifies empirical risk minimization and the use of a validation set to estimate the loss outside the training set. Each requirement can be tested. Independence can be tested through its consequence for predictability of individual data points. A common probability distribution can be tested by the largest gap between the empirical cdfs of two blocks of the sequence. The assumption fails under distribution shift, when the data points met after training follow a different probability distribution.

## Definition

A weather station at Krems an der Donau recorded the maximum air temperature of each day throughout 2024. Fig. 1 shows four of those months, each a dataset with one number per day. The numbers of one such month can be viewed as realizations of independent and identically distributed RVs. That interpretation is the i.i.d. assumption, and the figure shows evidence against it. The four months have different averages. Inside every month a warm day is often followed by a warm one, so which day is read decides what the neighboring values are.

![Figure 1 of the entry iidasspt](https://dictionaryofml.org/terms/fig/iidasspt-fig1.svg)

*Figure 1: The maximum air temperature of each day at Krems an der Donau, for four months of 2024. Each month is read as one dataset. The averages differ from month to month, $5.95$ degrees in January against $30.09$ in August, and within each month a warm day tends to follow a warm one. Data generated by pythondemos/iidasspt.py*

The i.i.d. assumption is the probabilistic model representing the generation of the data points ${\mathbf z}^{(1)}, \ldots,
{\mathbf z}^{(m)} \in \mathcal{Z}$ of a dataset by independent and identically distributed RVs with one unknown probability distribution $p({\mathbf z})$ (Shalev-Shwartz and Ben-David, 2014, Sect. 2.3.1). For any events $\mathcal{A}_{1}, \ldots, \mathcal{A}_{m} \subseteq
\mathcal{Z}$,

\begin{equation}
\tag{1}
\mathbb{P}\left( {\mathbf z}^{(1)} \in \mathcal{A}_{1}, \ldots,
{\mathbf z}^{(m)} \in \mathcal{A}_{m}\right)
= \prod_{r=1}^{m}
\mathbb{P}\left( {\mathbf z}^{(r)} \in \mathcal{A}_{r}\right)
\text{,}
\end{equation}

and each factor is computed from the same $p({\mathbf z})$. The assumption therefore consists of two separate requirements: the data points follow one common probability distribution, and they are jointly independent. Joint independence is stronger than independence of every pair, since the product rule Eq. (1) is required for all $m$ events at once: there are collections in which every pair factorizes and the whole collection does not (Murphy, 2012, Exercise 2.7).

The assumption names a quantity to aim at beyond the training set: the risk of a hypothesis, its expected loss $\mathbb{E} \big\{ L\left({\mathbf z},h \right) \big\}$ on a data point drawn from $p({\mathbf z})$. Under the i.i.d. assumption the average loss on a training set is an unbiased estimate of it (Shalev-Shwartz and Ben-David, 2014, Sect. 4.2). Let $\sigma^{2}$ be the variance of the loss of a fixed hypothesis on one data point. The average loss over $m$ data points then has variance $\sigma^{2}/m$, and for losses confined to an interval $[a,b]$, Hoeffding's inequality bounds the probability that it deviates from the risk by more than $\epsilon$ by $2 \exp \big( -2 m \epsilon^{2}/(b-a)^{2} \big)$ (Hoeffding, 1963; Shalev-Shwartz and Ben-David, 2014, Lemma B.6). This is the argument for empirical risk minimization and for using a validation set to estimate the loss outside the training set (Shalev-Shwartz and Ben-David, 2014, Thm. 11.1).

The assumption can fail: a distribution shift violates the identically distributed requirement across datasets, since the probability distribution that generated the training set differs from the one that generates the data points encountered after deployment (Quiñonero-Candela et al., 2009). Verifying that a given dataset can be approximated by realizations of i.i.d. RVs has two parts, each a form of hypothesis testing. A test first examines whether the data points are realizations of independent RVs. A second test then examines whether those RVs share one probability distribution.

One basic independence test uses the consequence that statistical independence has for the predictability of a data point from a few preceding ones. If ${\mathbf z}^{(1)}, \ldots, {\mathbf z}^{(d)},
{\mathbf z}^{(d+1)}$ are realizations of independent and identically distributed RVs, then no hypothesis predicts ${\mathbf z}^{(d+1)}$ from ${\mathbf z}^{(1)}, \ldots, {\mathbf z}^{(d)}$.

The hypothesis is learned from data points constructed out of the sequence: each is an overlapping block of $d+1$ consecutive values, whose first $d$ values are its features and whose last value is its label. Learning by empirical risk minimization on part of the blocks and measuring the average loss on the rest gives the quantity to compare. If the values are realizations of independent RVs, that loss is similar to the loss of a hypothesis that uses no features at all, namely the best constant. Any hypothesis space serves: a linear model with $d = 1$ under the squared error loss, learned and judged on the same blocks, lowers the average loss below that of the best constant by the factor $1 - r^{2}$, where $r$ is the autocorrelation at lag one, the Pearson correlation coefficient between consecutive values. The factor follows from least squares: the slope is $r$ times the ratio of the standard deviations of label and feature, and the average squared residual is then $1 - r^{2}$ times the sample variance of the labels. A decision tree, in turn, registers dependence that no Pearson correlation coefficient can.

Fig. 2 runs this on the Krems record. A decision tree of depth three with the preceding days as features reaches an average loss of $11.8$ on days it was not fitted on, against the $84.6$ of the best constant, so the preceding days carry information about the next one and the data points are not realizations of independent RVs.

![Figure 2 of the entry iidasspt](https://dictionaryofml.org/terms/fig/iidasspt-fig2.svg)

*Figure 2: (a) Sixty days of the Krems an der Donau record and the predictions of a decision tree of depth three reading the $d$ preceding days. (b) The average loss of that decision tree on days it was not fitted on, against $d$, with the loss of the best constant as the dotted line. Using the preceding days cuts the loss to about one eighth of it. Data generated by pythondemos/iidasspt.py*

The check detects only the dependence that its hypothesis space and its label can express. A sequence can be dependent and still have a flat conditional expectation, and then no hypothesis space helps. For $x^{(r)} = \varepsilon^{(r)}
\varepsilon^{(r-1)}$ with independent and identically distributed standard Gaussian RVs $\varepsilon$, decision trees of depth one through four all predict the next value worse than the best constant does, and monotonically worse the deeper the tree. Enlarging the hypothesis space lowers no loss when there is nothing in the conditional expectation to find, and worsens generalization: the larger hypothesis space only lowers the loss on the days the hypothesis was trained on. Predicting the squared value instead brings the loss below the constant's. Enlarging the hypothesis space therefore detects dependence only where it appears in the conditional expectation; changing what the hypothesis is asked to predict detects it elsewhere. Neither alone turns the check into a test of independence (see autocorrelation).

The identically distributed requirement needs to be verified in turn. Cut the dataset into blocks of equal size and compare the empirical distributions of two blocks by the largest vertical gap between their cdfs. That gap is the two-sample Kolmogorov–Smirnov statistic (Gretton et al., 2012, Sect. 7.2.1). It is at most one, and it equals one exactly when every value of one block lies below every value of the other.

The Krems record fails this requirement too, and by a wide margin. Fig. 3 applies the block comparison over periods of growing length. Within January the largest gap between the two $15$-day blocks is $0.33$; over January to June and over the whole year it is $1$, so every value of one of the two blocks lies below every value of the other.

![Figure 3 of the entry iidasspt](https://dictionaryofml.org/terms/fig/iidasspt-fig3.svg)

*Figure 3: Empirical cdfs of the two $15$-day blocks that are furthest apart, for three periods of the Krems an der Donau record. Data generated by pythondemos/iidasspt.py*

For two blocks of $n$ values each, i.i.d. RVs produce a gap above $\sqrt{\ln(2.17/\alpha)/n}$ with probability at most $\alpha$, by the two-sample form of the Dvoretzky–Kiefer–Wolfowitz inequality (Wei and Dudley, 2012, Thm. 1). The constant $2.17$ is what two blocks of equal size $n \geq 4$ require. The value $2$, tight for the one-sample inequality (Massart, 1990, p. 1270), holds in the two-sample case only from $n = 458$ on (Wei and Dudley, 2012, Thm. 1). For $n = 15$ and $\alpha = 0.05$ the threshold is $0.50$, which the January gap of $0.33$ does not reach. A gap below its threshold proves nothing beyond itself: the blocks then carry no evidence against a common probability distribution, while a dependence between consecutive values can still be present. This is why both parts are needed.

A cruder check compares monthly averages: January averages $5.95$ degrees and July $28.99$. Take the common probability distribution to be Gaussian, with its expectation and variance set to the sample mean $17.91$ and the sample variance of the whole year, a standard deviation of $9.21$ degrees. The difference between two monthly averages of $31$ days each then has expectation zero and standard deviation $2.34$ degrees, so the observed gap of $23.03$ degrees lies $9.8$ standard deviations from zero, which that probability distribution produces with probability below $10^{-21}$. Subtracting the seasonal average removes this violation, and the dependence survives it.

**See also:** [independent and identically distributed](https://dictionaryofml.org/terms/iid.html), autocorrelation, Ljung–Box test, probabilistic model, [data point](https://dictionaryofml.org/terms/datapoint.html), [random variable](https://dictionaryofml.org/terms/rv.html), risk, [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), [training set](https://dictionaryofml.org/terms/trainset.html), [validation set](https://dictionaryofml.org/terms/valset.html), [distribution shift](https://dictionaryofml.org/terms/distshift.html), empirical distribution, hypothesis testing, [hypothesis space](https://dictionaryofml.org/terms/hypospace.html), [decision tree](https://dictionaryofml.org/terms/decisiontree.html), realization.

## References

- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Murphy (2012). Machine Learning: A Probabilistic Perspective. *MIT Press*.
- Hoeffding (1963). Probability Inequalities for Sums of Bounded Random Variables. *J. Amer. Statistical Assoc.*. <https://doi.org/10.1080/01621459.1963.10500830>
- Quiñonero-Candela et al. (2009). Dataset Shift in Machine Learning. *MIT Press*.
- Gretton et al. (2012). A Kernel Two-Sample Test. *Journal of Machine Learning Research*. <https://www.jmlr.org/papers/v13/gretton12a.html>
- Wei and Dudley (2012). Two-sample Dvoretzky--Kiefer--Wolfowitz inequalities. *Statist. Probab. Lett.*. <https://doi.org/10.1016/j.spl.2011.11.012>
- Massart (1990). The Tight Constant in the Dvoretzky--Kiefer--Wolfowitz Inequality. *Ann. Probab.*.

## Cite this entry

```bibtex
@misc{dictml_iidasspt,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {independent and identically distributed assumption (i.i.d. assumption)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/iidasspt.html}
}
```


---

# inner product

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/innerproduct.html) · [typeset PDF](https://dictionaryofml.org/terms/innerproduct.pdf) · [LaTeX](https://dictionaryofml.org/terms/innerproduct.tex) · [Python demo](https://dictionaryofml.org/terms/innerproduct.py) · [notebook](https://dictionaryofml.org/terms/innerproduct.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

The inner product of two vectors in a vector space is a number that measures their similarity. Many machine learning applications represent a data point by a numeric array of features, such as the color intensities of an image, the amplitudes of a sensor signal, or the embeddings of the tokens used by LLMs. Each such array is a vector in some vector space. Many machine learning methods rely on measuring the similarity between the vectors that represent data points. Each neuron of an artificial neural network matches its input against a learned template, and the attention unit of a large language model compares the query vector of one token with the key vector of another. Both comparisons compute an inner product.

## Definition

The color intensities of the pixels of an image, the amplitudes of a sensor signal at regular intervals, and the embedding of a token in a large language model are numeric arrays, and data points are often characterized by features that form such an array. Such an array can be represented by a vector, i.e., an element of a vector space. Many machine learning methods compare two data points by measuring how similar the vectors that represent them are. One such measure of similarity is an inner product.

Consider a vector space $\mathcal{V}$ over the field of real numbers $\mathbb{R}$. An inner product in $\mathcal{V}$ is a function

$$
\langle \cdot, \cdot \rangle: \mathcal{V} \times \mathcal{V} \to \mathbb{R}
$$

that satisfies the following properties for all vectors ${\mathbf u}, {\mathbf v}, {\mathbf w} \in \mathcal{V}$ and all scalars $\beta \in \mathbb{R}$ (Axler, 2024, Def. 6.2):

- Symmetry: $\langle {\mathbf u}, {\mathbf v} \rangle = \langle {\mathbf v}, {\mathbf u} \rangle$;
- Linearity in the first argument: $\langle \beta {\mathbf u} + {\mathbf w}, {\mathbf v} \rangle = \beta \langle {\mathbf u}, {\mathbf v} \rangle + \langle {\mathbf w}, {\mathbf v} \rangle$;
- Positive-definiteness: $\langle {\mathbf u}, {\mathbf u} \rangle \geq 0$, with equality if and only if ${\mathbf u} = \mathbf{0}$.

Over the field of complex numbers, symmetry becomes conjugate symmetry, $\langle {\mathbf u}, {\mathbf v} \rangle =
\overline{\langle {\mathbf v}, {\mathbf u} \rangle}$, and the inner product is conjugate-linear in the second argument (Axler, 2024, Thm. 6.6); only real vector spaces are considered from here on. Two vectors ${\mathbf u}, {\mathbf v} \in \mathcal{V}$ are called orthogonal if their inner product is zero, $\langle {\mathbf u}, {\mathbf v} \rangle = 0$.

The pair $(\mathcal{V}, \langle \cdot, \cdot \rangle)$ is called an inner product space. Each inner product induces a norm via $\Vert {{\mathbf u}} \Vert := \sqrt{\langle {\mathbf u}, {\mathbf u} \rangle}$ for all ${\mathbf u} \in \mathcal{V}$ (Axler, 2024, Thm. 6.17), which in turn induces a metric via $d({\mathbf u},{\mathbf v}) := \Vert {{\mathbf u} - {\mathbf v}} \Vert$ for all ${\mathbf u}, {\mathbf v} \in \mathcal{V}$.

For the Euclidean space $\mathcal{V} = \mathbb{R}^{d}$, the standard inner product, also called the dot product, is $\langle {\mathbf x}, {\mathbf x}' \rangle = {\mathbf x}^{\top}
{\mathbf x}' = \sum_{j=1}^{d} x_{j} x'_{j}$ (Strang, 2016, Sect. 1.2). Geometrically, the inner product determines orthogonal projections: the orthogonal projection of ${\mathbf x}'$ onto the direction of a nonzero vector ${\mathbf x}$ has the signed length $\langle {\mathbf x}, {\mathbf x}' \rangle / \Vert {{\mathbf x}} \Vert$ (Axler, 2024, Example 6.56) (see Fig. 1).

![Figure 1 of the entry innerproduct](https://dictionaryofml.org/terms/fig/innerproduct-fig1.svg)

*Figure 1: Geometry of the standard inner product in the Euclidean space $\mathbb{R}^{2}$. The dashed ruler drops from ${\mathbf x}'$ perpendicularly onto the direction of ${\mathbf x}$ and meets that direction at the orthogonal projection of ${\mathbf x}'$. This orthogonal projection lies at the signed length $\langle {\mathbf x}, {\mathbf x}' \rangle / \Vert {{\mathbf x}} \Vert$ from the origin, marked by the double arrow. The angle $\theta$ between the two vectors satisfies $\cos \theta = \langle {\mathbf x}, {\mathbf x}' \rangle / \big( \Vert {{\mathbf x}} \Vert \Vert {{\mathbf x}'} \Vert \big)$ (defined below). The inner product measures the alignment of the two vectors and vanishes when they are orthogonal*

The inner product also defines the angle $\theta$ between two nonzero vectors ${\mathbf x}, {\mathbf x}' \in \mathcal{V}$, via $\cos \theta := \langle {\mathbf x}, {\mathbf x}' \rangle / \big( \Vert {{\mathbf x}} \Vert \Vert {{\mathbf x}'} \Vert \big)$. The Cauchy-Schwarz inequality $|\langle {\mathbf x}, {\mathbf x}' \rangle| \leq \Vert {{\mathbf x}} \Vert \Vert {{\mathbf x}'} \Vert$ ensures that this ratio lies in $[-1, 1]$ (Axler, 2024, Thm. 6.14). Here $\cos$ is the cosine function, which decreases strictly from $\cos 0 = 1$ to $\cos \pi = -1$, since its derivative $-\sin$ is negative on $(0, \pi)$, and therefore takes each value of $[-1,1]$ exactly once on the interval $[0, \pi]$. For the Euclidean plane $\mathbb{R}^{2}$, as illustrated by Fig. 1, the angle defined in this way agrees with the elementary geometric angle between two vectors (Strang, 2016, Sect. 1.2).

For two vectors with given norms, the sign and magnitude of their inner product are determined by the angle $\theta$ between them. The inner product is positive for $\theta < \pi/2$, zero for $\theta = \pi/2$, where the two vectors are orthogonal, and negative for $\theta > \pi/2$. The angle drawn in Fig. 1 is smaller than $\pi/2$: the two vectors have a positive inner product, and the orthogonal projection therefore falls on the same side of the origin as ${\mathbf x}$.

Fig. 1 projects one vector onto the direction of another, that is, onto the line it spans. The same characterization holds for a whole subspace, and it is what turns approximation problems, with respect to the induced norm, into systems of linear equations. Consider a subspace $\mathcal{U} \subseteq \mathcal{V}$ and a vector ${\mathbf v} \in \mathcal{V}$. In general, the vector of $\mathcal{U}$ closest to ${\mathbf v}$ with respect to some norm is called the projection of ${\mathbf v}$ onto $\mathcal{U}$. If the norm is induced by some inner product, the projection becomes an orthogonal projection. A vector $\widehat{{\mathbf v}} \in \mathcal{U}$ minimizes $\Vert {{\mathbf v} - {\mathbf u}} \Vert$ over all ${\mathbf u} \in \mathcal{U}$ if and only if the approximation error ${\mathbf v} - \widehat{{\mathbf v}}$ is orthogonal to every vector in $\mathcal{U}$,

\begin{equation}
\tag{1}
\langle {\mathbf v} - \widehat{{\mathbf v}}, {\mathbf u} \rangle = 0
\quad \text{for all } {\mathbf u} \in \mathcal{U}
\text{.}
\end{equation}

The reason is the Pythagorean identity

$$
\Vert {{\mathbf v} - {\mathbf u}} \Vert^{2} = \Vert {{\mathbf v} - \widehat{{\mathbf v}}} \Vert^{2}
+ \Vert {\widehat{{\mathbf v}} - {\mathbf u}} \Vert^{2}
\text{,}
$$

which holds for every ${\mathbf u} \in \mathcal{U}$ whenever Eq. (1) does. The second term is non-negative, so no vector of $\mathcal{U}$ is closer to ${\mathbf v}$ than $\widehat{{\mathbf v}}$ is: orthogonality of the error and being closest are the same condition (Axler, 2024, Thm. 6.61). The orthogonality condition Eq. (1) is linear in $\widehat{{\mathbf v}}$, so a minimizer can be computed by solving linear equations.

Fig. 2 illustrates inner products between feature vectors in $\mathbb{R}^{2}$ that represent five European cities. Each city is characterized by its latitude and longitude as its two features, and drawn at the vector $x_{1} {\mathbf u} + x_{2} {\mathbf v}$ for a fixed orthonormal pair ${\mathbf u}, {\mathbf v} \in \mathbb{R}^{2}$. The orthogonal projections of the other cities onto the direction of Helsinki rank them by their alignment with it.

![Figure 2 of the entry innerproduct](https://dictionaryofml.org/terms/fig/innerproduct-fig2.svg)

*Figure 2: Five European cities, each represented by a feature vector ${\mathbf x} = (x_{1}, x_{2})^{\top} \in \mathbb{R}^{2}$ with latitude $x_{1}$ and longitude $x_{2}$ as features. Each city is drawn at the vector $x_{1} {\mathbf u} + x_{2} {\mathbf v}$ (filled circles) for a fixed orthonormal pair ${\mathbf u}, {\mathbf v} \in \mathbb{R}^{2}$, chosen such that the vector of Helsinki (thick arrow) lies on the horizontal line and the other cities lie above it. The drawn arrows are therefore not the feature vectors $(x_{1}, x_{2})^{\top}$ themselves. However, since ${\mathbf u}, {\mathbf v}$ are orthonormal, the inner product of two vectors equals the standard inner product of the corresponding feature vectors, $\langle x_{1} {\mathbf u} + x_{2} {\mathbf v}, x_{1}' {\mathbf u} + x_{2}' {\mathbf v} \rangle = x_{1} x_{1}' + x_{2} x_{2}'$. Each dashed ruler projects the vector of a city orthogonally onto the Helsinki vector. The open circles mark the resulting orthogonal projections. City coordinates retrieved from OpenStreetMap*

The characterization extends beyond subspaces. For a non-empty closed convex set $\mathcal{C} \subseteq \mathbb{R}^{d}$, a vector $\widehat{{\mathbf v}} \in \mathcal{C}$ is the projection of ${\mathbf v}$ onto $\mathcal{C}$ if and only if

\begin{equation}
\tag{2}
\langle {\mathbf v} - \widehat{{\mathbf v}}, {\mathbf u} - \widehat{{\mathbf v}} \rangle \leq 0
\quad \text{for all } {\mathbf u} \in \mathcal{C}
\text{.}
\end{equation}

Such a projection exists and is unique (Bertsekas, 2009, Prop. 1.1.9).

One application of the subspace case Eq. (1) is a characterization of linear regression. The learned model parameters are those for which the prediction error is orthogonal to every column of the feature matrix (see normal equations). The resulting predictions are therefore the orthogonal projection of the label vector onto the column space of the feature matrix (Hastie et al., 2009, Sect. 3.2). The convex case Eq. (2) becomes relevant when linear regression is modified by adding a convex constraint on the model parameters.

The Euclidean space $\mathbb{R}^{d}$ carries other inner products besides the standard one (Axler, 2024, Example 6.3). For example, each symmetric and positive definite matrix ${\mathbf A} \in \mathbb{R}^{d \times d}$, i.e., one with ${\mathbf x}^{\top} {\mathbf A} {\mathbf x} > 0$ for every nonzero ${\mathbf x}$, induces an inner product $\langle {\mathbf x}, {\mathbf x}' \rangle := {\mathbf x}^{\top} {\mathbf A} {\mathbf x}'$. Such a matrix weighs the coordinate directions against each other, and thereby fixes a different notion of similarity between data points. Take ${\mathbf A} = \operatorname{diag}(4, 1/4)$, which counts the first feature sixteen times as heavily as the second, and the query ${\mathbf x} = (1, 1)^{\top}$. Of the two candidates $(0.9, 1.2)^{\top}$ and $(1.3, 0.4)^{\top}$, the standard inner product is larger for the first ($2.1$ against $1.7$) and the weighted one larger for the second ($5.3$ against $3.9$): the same query and the same candidates, but a different answer to which is more similar. Since an inner product also induces a norm and a metric, the choice of ${\mathbf A}$ fixes which data points count as nearest neighbors as well.

An inner product also gives rise to the notion of an orthonormal basis. Consider a basis ${\mathbf b}^{(1)}, \ldots, {\mathbf b}^{(d)}$ of a vector space $\mathcal{V}$ of dimension $d$. It is orthonormal if its elements are pairwise orthogonal and have unit norm, $\langle {\mathbf b}^{(j)}, {\mathbf b}^{(j')} \rangle = 0$ for $j \neq j'$ and $\langle {\mathbf b}^{(j)}, {\mathbf b}^{(j)} \rangle = 1$ (Axler, 2024, Def. 6.22). The coordinates of a vector with respect to an orthonormal basis are delivered by inner products: ${\mathbf u} = \sum_{j=1}^{d} \beta_{j} {\mathbf b}^{(j)}$ with expansion coefficients $\beta_{j} = \langle {\mathbf u}, {\mathbf b}^{(j)} \rangle$ (Axler, 2024, Thm. 6.30). Conversely, an inner product can be defined by declaring a basis orthonormal. Take any basis ${\mathbf b}^{(1)}, \ldots, {\mathbf b}^{(d)}$ of $\mathcal{V}$. Then setting

$$
\langle {\mathbf u}, {\mathbf v} \rangle := \sum_{j=1}^{d} \beta_{j} \beta'_{j}
\quad \text{for } {\mathbf u} = \sum_{j=1}^{d} \beta_{j} {\mathbf b}^{(j)}
\text{ and } {\mathbf v} = \sum_{j=1}^{d} \beta'_{j} {\mathbf b}^{(j)}
$$

yields the unique inner product for which this basis is orthonormal. The standard inner product of the Euclidean space arises in this way from the standard basis given by the columns of the identity matrix $\mathbf{I}_{d}$.

Several machine learning constructions use the similarity measure of the opening paragraph. An embedding maps a text block to a vector in a vector space. The alignment of two text blocks is measured by an inner product of their corresponding embeddings (Goodfellow et al., 2016, Sect. 12.4.2). Each neuron of an artificial neural network matches its input vector against a vector of learned model parameters (Goodfellow et al., 2016, Ch. 6). Attention scores in a large language model are scaled inner products between query and key vectors (Vaswani et al., 2017).

Many machine learning methods access the feature vectors of data points only through inner products between feature vectors. A linear model uses hypotheses $h({\mathbf x}) = \langle {\mathbf w}, {\mathbf x} \rangle$. Consider regularized empirical risk minimization over a training set $\mathcal{D}^{(\rm train)} = \big\{ \big({\mathbf x}^{(r)}, y^{(r)}\big) \big\}_{r=1}^{m}$, with a loss that depends on ${\mathbf w}$ only through the predictions $\langle {\mathbf w}, {\mathbf x}^{(r)} \rangle$ and a penalty that increases strictly with $\Vert {{\mathbf w}} \Vert$. Every minimizer is a linear combination of the feature vectors in the training set, $\widehat{{\mathbf w}} = \sum_{r=1}^{m} \beta_{r} {\mathbf x}^{(r)}$. This is the representer theorem (Schölkopf and Smola, 2002, Thm. 4.2), and the support-vector expansion of the support vector machine is a special case of it (Lampert, 2009, Thm. 2.3).

The prediction for a new data point then reads

$$
\widehat{h}({\mathbf x}) = \sum_{r=1}^{m}
\beta_{r} \langle {\mathbf x}^{(r)}, {\mathbf x} \rangle
\text{.}
$$

Training and prediction therefore use the feature vectors only through inner products. Within the training set these are $\langle {\mathbf x}^{(r)}, {\mathbf x}^{(r')} \rangle$, and between the training set and the new data point they are $\langle {\mathbf x}^{(r)}, {\mathbf x} \rangle$. This is used by kernel methods that use a kernel $k({\mathbf x}, {\mathbf x}')$ to compute inner products without ever forming feature vectors (Schölkopf and Smola, 2002) (see kernel method).

Methods based on Euclidean distances access the feature vectors only through inner products as well, even when they are defined without reference to one. The squared Euclidean distance between two feature vectors is a sum of three inner products, $\left\lVert {\mathbf x} - {\mathbf x}' \right\rVert_{2}^{2} = \langle {\mathbf x}, {\mathbf x} \rangle - 2 \langle {\mathbf x}, {\mathbf x}' \rangle + \langle {\mathbf x}', {\mathbf x}' \rangle$. A method that uses the feature vectors only through their Euclidean distances therefore also uses them only through inner products. k-nearest neighbors is one example: it ranks the data points in the training set by the distance of their feature vectors to the feature vector of the new data point (Hastie et al., 2009, Sect. 2.3.2). The same kernel substitution therefore applies to it: by the identity above, each Euclidean distance is computed from three kernel evaluations, and no feature vector is ever formed.

**Synonyms:** scalar product, dot product.

**See also:** [norm](https://dictionaryofml.org/terms/norm.html), vector, [Hilbert space](https://dictionaryofml.org/terms/hilbertspace.html), orthogonality condition, Euclidean space, Cauchy-Schwarz inequality, orthogonal projection, kernel.

## References

- Axler (2024). Linear Algebra Done Right. *Springer*. <https://doi.org/10.1007/978-3-031-41026-0>
- Strang (2016). Introduction to Linear Algebra. *Wellesley-Cambridge Press*.
- Bertsekas (2009). Convex Optimization Theory. *Athena Scientific*.
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Goodfellow et al. (2016). Deep Learning. *MIT Press*.
- Vaswani et al. (2017). Attention is all you need. *Adv. Neural Inf. Process. Syst.*. <https://papers.nips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html>
- Schölkopf and Smola (2002). Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. *MIT Press*. <https://doi.org/10.7551/mitpress/4175.001.0001>
- Lampert (2009). Kernel Methods in Computer Vision. *Found. Trends Comput. Graph. Vis.*. <https://doi.org/10.1561/0600000027>

## Cite this entry

```bibtex
@misc{dictml_innerproduct,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {inner product},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/innerproduct.html}
}
```


---

# interpretability

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-10-29. [HTML](https://dictionaryofml.org/terms/interpretability.html) · [typeset PDF](https://dictionaryofml.org/terms/interpretability.pdf) · [LaTeX](https://dictionaryofml.org/terms/interpretability.tex) · [Python demo](https://dictionaryofml.org/terms/interpretability.py) · [notebook](https://dictionaryofml.org/terms/interpretability.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

The interpretability of a machine learning method is the extent to which a human user can comprehend the computational process it carries out. Interpretability is closely related to predictability: the user should be able to anticipate the predictions of the method on a test set. Interpretability differs from explainability, which concerns understanding specific predictions with the help of provided explanations.

## Definition

Consider a training set of monthly sales and a linear hypothesis learned from it. Computing next month's prediction from that hypothesis takes a slope, an intercept, and one multiplication, and a user can carry the computation out on paper. With $d$ features, the fitted linear map has one weight per feature and the prediction takes $d$ multiplications, still a computation the user can carry out. The same prediction delivered by a deep net arrives with no account a user could follow. Interpretability is what separates the two.

A machine learning method is interpretable for a human user if they can comprehend the computational process of the method. The international terminology standard for artificial intelligence does not define interpretability (International Organization for Standardization and International Electrotechnical Commission, 2022). Its closest notion is predictability, the property of an artificial intelligence system that enables reliable assumptions by users about the predictions delivered by a machine learning method (International Organization for Standardization and International Electrotechnical Commission, 2022, Sect. 3.5.8 and 5.15.7).

Comprehension of a computation cannot be observed directly, so interpretability is judged through predictability: the user anticipates the predictions the method delivers, and the evaluations proposed in the literature measure how often those anticipations are right (Chen et al., 2018; Colin et al., 2022; Doshi-Velez and Kim, 2017; Hase and Bansal, 2020). Predictability is weaker than interpretability: a method whose computation the user can follow is predictable to that user, but a method whose predictions the user anticipates need not be one whose computation they could carry out. The terminology standard makes the same point: a user may rely on the predictions of an artificial intelligence system without being able to comprehend how the method produced them (International Organization for Standardization and International Electrotechnical Commission, 2022, Sect. 5.15.7).

Fig. 1 depicts a test of predictability. A training set is used by two different methods that learn hypotheses $\widehat{h}$ and $\widehat{h}'$. The user anticipates the predictions on a test set, providing a pseudo-label for each of its data points, by visually extrapolating the linear trend of the training set. The predictions of $\widehat{h}$ agree with these anticipations: the method producing $\widehat{h}$ is predictable to the user. The predictions of $\widehat{h}'$ deviate from them: the method producing $\widehat{h}'$ fails the test of predictability and therefore cannot be one whose computation the user comprehends. The agreement alone does not establish interpretability, since the user anticipated the predictions of $\widehat{h}$ without carrying out its computation. The method producing $\widehat{h}$ is nevertheless also interpretable to a user familiar with the concept of a linear map: such a user could comprehend the computation of $\widehat{h}$.

![Figure 1 of the entry interpretability](https://dictionaryofml.org/terms/fig/interpretability-fig1.svg)

*Figure 1: A human user provides pseudo-labels (filled red squares) for the test set $\mathcal{D}^{(\rm test)}$ by visually extrapolating the linear trend of the training set $\mathcal{D}^{(\rm train)}$ (filled blue circles); the vertical double arrows mark the prediction errors of $\widehat{h}$ on the training set. The predictions of the hypothesis $\widehat{h}$ agree with these pseudo-labels, while the predictions of another hypothesis $\widehat{h}'$ deviate from them. The user anticipates the predictions here without carrying out the computation (a slope, an intercept, and one multiplication), which would be required to comprehend the method*

Besides following the computation step by step, as for the linear map of the sales example, a user can comprehend a learned hypothesis through its decomposition into components that are themselves interpretable (Lipton, 2018). The hypothesis $\widehat{h}$ in Fig. 1 decomposes into a slope and an intercept: the slope states how the prediction changes with the feature. A trained deep net lacks such a decomposition, since the activation of an individual neuron typically does not correspond to a human-interpretable concept. Mechanistic interpretability aims to recover interpretable components from a trained deep net (Elhage et al., 2021; Olah et al., 2020) by decomposition methods such as sparse coding of activations (Cunningham et al., 2024).

The AI Risk Management Framework of the US National Institute of Standards and Technology distinguishes interpretability from explainability (National Institute of Standards and Technology, 2023), the property pursued by explainable artificial intelligence (Gunning and Aha, 2019). In contrast to interpretability, explainability requires that an explanation is provided along with each prediction. These explanations may take the form of saliency maps or reference examples from the training set. A challenge for explainable artificial intelligence methods is that a computed explanation can be unfaithful to the learned hypothesis and thereby mislead the user. An interpretable machine learning method avoids this: the account of its computation is the computation itself, so the account cannot be unfaithful to it (Rudin, 2019).

**See also:** [explainability](https://dictionaryofml.org/terms/explainability.html), [explainable artificial intelligence](https://dictionaryofml.org/terms/xaiterm.html), [interpretable machine learning](https://dictionaryofml.org/terms/interpretableml.html), [trustworthy artificial intelligence](https://dictionaryofml.org/terms/trustAI.html), [regularization](https://dictionaryofml.org/terms/regularization.html), local interpretable model-agnostic explanations, mechanistic interpretability, [transparency](https://dictionaryofml.org/terms/transparency.html).

## References

- International Organization for Standardization and International Electrotechnical Commission (2022). ISO/IEC 22989:2022 --- Information technology --- Artificial intelligence --- Artificial intelligence concepts and terminology. <https://www.iso.org/standard/74296.html>
- Chen et al. (2018). Learning to Explain: An Information-Theoretic Perspective on Model Interpretation. *Proc. 35th Int. Conf. Mach. Learn.*. <https://proceedings.mlr.press/v80/chen18j.html>
- Colin et al. (2022). What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods. *Adv. Neural Inf. Process. Syst.*. <https://proceedings.neurips.cc/paper_files/paper/2022/hash/13113e938f2957891c0c5e8df811dd01-Abstract-Conference.html>
- Doshi-Velez and Kim (2017). Towards a rigorous science of interpretable machine learning. *arXiv preprint arXiv:1702.08608*. <https://arxiv.org/abs/1702.08608>
- Hase and Bansal (2020). Evaluating explainable AI: Which algorithmic explanations help users predict model behavior?. *Proc. 58th Annu. Meeting Assoc. Comput. Linguistics*. <https://aclanthology.org/2020.acl-main.491>
- Lipton (2018). The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery. *Queue*. <https://doi.org/10.1145/3236386.3241340>
- Elhage et al. (2021). A Mathematical Framework for Transformer Circuits. *Transformer Circuits Thread*. <https://transformer-circuits.pub/2021/framework/index.html>
- Olah et al. (2020). Zoom In: An Introduction to Circuits. *Distill*. <https://doi.org/10.23915/distill.00024.001>
- Cunningham et al. (2024). Sparse Autoencoders Find Highly Interpretable Features in Language Models. *12th Int. Conf. Learn. Representations*. <https://openreview.net/forum?id=F76bwRSLeK>
- National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). <https://doi.org/10.6028/NIST.AI.100-1>
- Gunning and Aha (2019). DARPA's Explainable Artificial Intelligence Program. *AI Magazine*. <https://doi.org/10.1609/aimag.v40i2.2850>
- Rudin (2019). Stop explaining black box machine learning models for high-stakes decisions and use interpretable models instead. *Nature Mach. Intell.*. <https://doi.org/10.1038/s42256-019-0048-x>

## Cite this entry

```bibtex
@misc{dictml_interpretability,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {interpretability},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/interpretability.html}
}
```


---

# interpretable machine learning (interpretable ML)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/interpretableml.html) · [typeset PDF](https://dictionaryofml.org/terms/interpretableml.pdf) · [LaTeX](https://dictionaryofml.org/terms/interpretableml.tex) · [Python demo](https://dictionaryofml.org/terms/interpretableml.py) · [notebook](https://dictionaryofml.org/terms/interpretableml.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Interpretable machine learning (interpretable ML) refers to machine learning methods whose hypothesis space contains only hypotheses that a human user can comprehend directly, such as sparse linear models or shallow decision trees. Such methods let a user see how a prediction changes when the features of a data point change, and how the predictions change when a different training set is used. An interpretable hypothesis is its own explanation; post hoc explanations of the predictions of an opaque learned hypothesis can instead be unfaithful. Restricting the hypothesis space to interpretable hypotheses also acts as a form of regularization. A penalty term can pick a comprehensible hypothesis out of a hypothesis space that mostly holds others: least absolute shrinkage and selection operator sets some weights of a linear model to exactly zero, how many depending on its regularization parameter, leaving a hypothesis with few nonzero weights, and explainable empirical risk minimization penalizes deviations from the predictions one user supplies during training.

## Definition

A physician reads a rule that predicts a patient's risk from a handful of measurements and can check every step of it by hand. Interpretable ML refers to machine learning methods whose hypothesis space contains only such hypotheses, which a human user can comprehend directly (see interpretability). Such methods let a user follow both what a learned hypothesis does to its input and how training produced it. An interpretable machine learning method allows the user to understand how a prediction changes when the features of a data point change, and how the predictions change when a different training set is used. Thus, interpretable machine learning methods require suitable choices for the hypothesis space and the training algorithm.

Examples of interpretable machine learning include empirical risk minimization with linear models that combine a small number of meaningful features, shallow decision trees whose predictions follow from a few explicit tests, GAMs, and lists of decision rules (Molnar, 2025). Fig. 1 depicts a decision tree for medical triage in a hospital. This system involves two tests of vital signs, allowing the staff to trace every prediction by reading the tree.

![Figure 1 of the entry interpretableml](https://dictionaryofml.org/terms/fig/interpretableml-fig1.svg)

*Figure 1: An interpretable hypothesis: a shallow decision tree for medical triage. Every prediction follows from at most two explicit tests of the features, so a human user can trace and anticipate the predictions*

An interpretable hypothesis makes the effect of feature changes explicit. For a linear model $h^{({\mathbf w})}({\mathbf x}) = {\mathbf w}^{\top}{\mathbf x}$, changing the feature $x_{j}$ by an amount $\Delta$ changes the prediction by exactly $w_{j} \Delta$. Each weight states how strongly one feature affects the prediction. For the decision tree of Fig. 1, a prediction changes only when a feature crosses one of the explicit test thresholds. A patient at $38.5\,^{\circ}\mathrm{C}$ with a heart rate of $110/\mathrm{min}$ is routine; a rise to $38.9\,^{\circ}\mathrm{C}$ leaves that prediction unchanged, while a rise to $39.5\,^{\circ}\mathrm{C}$ turns it urgent.

An interpretable machine learning method also makes the effect of training set changes traceable. This dependence is studied under stability, which formalizes a machine learning method as a map from a training set to a learned hypothesis. For linear regression, this map is available in closed form through the normal equations whenever the feature matrix has full column rank (Hastie et al., 2009, Sect. 3.2, Eq. (3.6)), so the user can compute how a perturbed label in the training set shifts the predictions (see linear regression).

An interpretable hypothesis is its own explanation: no separate explanation is constructed for it. The explanation delivered with a prediction can then be a description of the learned hypothesis itself: the weights of a linear model that combines few features, or the flow chart of a shallow decision tree as in Fig. 1. A hypothesis space containing only such hypotheses is called intrinsically interpretable (Molnar, 2025). This contrasts with explainable artificial intelligence, which constructs post hoc explanations for the predictions of a possibly opaque learned hypothesis. For high-stakes applications, such as the medical triage of Fig. 1, using an interpretable model can be preferable to explaining an opaque model, since post hoc explanations can be unfaithful to the hypothesis they explain (Rudin, 2019).

Restricting $\mathcal{H}$ to interpretable hypotheses is model pruning, one of the three routes that regularization distinguishes. The penalty-term route gives an interpretable learned hypothesis without restricting the hypothesis space beforehand. A penalty term added to the objective function of empirical risk minimization penalizes the hypotheses a user cannot follow, so training picks a comprehensible one out of a hypothesis space that mostly holds the others. Least absolute shrinkage and selection operator does this for linear models, which are no more comprehensible than a deep net once they combine hundreds of features. Least absolute shrinkage and selection operator uses the $\ell_{1}$-norm $\left\lVert {\mathbf w} \right\rVert_{1}$ of the weights ${\mathbf w}$ as the penalty term, which sets some weights to exactly zero, how many depending on the regularization parameter $\alpha$ (Hastie et al., 2009, Sect. 3.4.2). The learned hypothesis is then a weighted sum of the remaining features, so a user can name the features the prediction uses and how much each one changes it.

A method close to this route, but not interpretable machine learning, is explainable empirical risk minimization, which extends the idea to the training of an arbitrary model. Its penalty term is built from the predictions that one specific user supplies for the data points of the training set (see explainable empirical risk minimization). It therefore delivers explainability to that user rather than interpretability, and it leaves the hypothesis space as it is. The user predictions need not be elicited one data point at a time. They might be obtained as the predictions of a simpler proxy model that the user considers interpretable, so that the penalty term measures the deviation from that proxy. In Fig. 2, the user predictions are obtained from a simple linear model fitted to the training set.

![Figure 2 of the entry interpretableml](https://dictionaryofml.org/terms/fig/interpretableml-fig2.svg)

*Figure 2: The idea of explainable empirical risk minimization (Zhang et al., 2024): training a high-dimensional model (polynomials of degree $10$) with a penalty term that measures the deviation of the predictions of the learned hypothesis from the user predictions. These user predictions are obtained from a simple surrogate linear model (dotted line). Empirical risk minimization without a penalty term (thin solid line) overfits the training set, while explainable empirical risk minimization delivers a hypothesis (dashed line) that stays close to the user predictions. Data generated by pythondemos/interpretableml.py*

**See also:** [interpretability](https://dictionaryofml.org/terms/interpretability.html), [explainability](https://dictionaryofml.org/terms/explainability.html), [explainable artificial intelligence](https://dictionaryofml.org/terms/xaiterm.html), [decision tree](https://dictionaryofml.org/terms/decisiontree.html), decision rule, generalized additive model, least absolute shrinkage and selection operator, stability, explainable empirical risk minimization.

## References

- Zhang et al. (2024). Explainable empirical risk minimization. *Neural Comput. Appl.*. <https://doi.org/10.1007/s00521-023-09269-3>
- Molnar (2025). Interpretable Machine Learning: A Guide for Making Black Box Models Explainable. *Ebook*. <https://christophm.github.io/interpretable-ml-book/>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Rudin (2019). Stop explaining black box machine learning models for high-stakes decisions and use interpretable models instead. *Nature Mach. Intell.*. <https://doi.org/10.1038/s42256-019-0048-x>

## Cite this entry

```bibtex
@misc{dictml_interpretableml,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {interpretable machine learning (interpretable ML)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/interpretableml.html}
}
```


---

# inverse

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/inverse.html) · [typeset PDF](https://dictionaryofml.org/terms/inverse.pdf) · [LaTeX](https://dictionaryofml.org/terms/inverse.tex) · [Python demo](https://dictionaryofml.org/terms/inverse.py) · [notebook](https://dictionaryofml.org/terms/inverse.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

An inverse of a function is a second function that undoes it. For $f: \mathcal{U} \to \mathcal{V}$ it is the function $g: \mathcal{V} \to \mathcal{U}$ with $g(f(u)) = u$ and $f(g(v)) = v$, and it exists precisely when $f$ is injective and every element of $\mathcal{V}$ has a nonempty preimage. The inverse of a linear map of $\mathbb{R}^{n}$ is again a linear map, represented by the inverse matrix ${\mathbf A}^{-1}$ with ${\mathbf A} {\mathbf A}^{-1} = {\mathbf A}^{-1} {\mathbf A} = \mathbf{I}$, which exists exactly when the square matrix ${\mathbf A}$ is of full-rank, equivalently when its determinant is nonzero. Where no inverse exists, a left or right inverse or the Moore–Penrose pseudoinverse takes its place. In machine learning, inverses appear in the closed-form solution of linear regression, in the pair formed by the sigmoid function and the logit, which turns a score into a probability and back, and in drawing a realization through the inverse of a cumulative distribution function.

## Definition

A frost warning for Krems an der Donau delivers a real-valued score for the day, and the sigmoid function turns that score into the probability of frost: the score $0$ becomes the probability $0.5$, the score $2.2$ becomes $0.9$. Reading the score back off a reported probability undoes this step, and the function that does so is the logit $\ln\big( p / (1-p) \big)$. The logit is the inverse of the sigmoid function.

An inverse of a function $f: \mathcal{U} \to \mathcal{V}$ is a function $g: \mathcal{V} \to \mathcal{U}$ that undoes it in both directions: $g\big(f(u)\big) = u$ for every $u \in \mathcal{U}$, and $f\big(g(v)\big) = v$ for every $v \in \mathcal{V}$. Such a $g$ exists precisely when $f$ is injective and every $v \in \mathcal{V}$ has a nonempty preimage. It is then unique and written $f^{-1}$. If only the first identity holds, $g$ is a left inverse, and if only the second holds, a right inverse. Fig. 1 shows both cases for a function on a domain of four shapes.

![Figure 1 of the entry inverse](https://dictionaryofml.org/terms/fig/inverse-fig1.svg)

*Figure 1: Two functions on the same domain $\mathcal{U}$ of four shapes, with $\mathcal{U}$ and the set $\mathcal{V}$ of values each drawn as a thin ellipse. In (a) the function $f$ sends a shape to its number of corners, and every value of $\mathcal{V} = \{0, 3, 4, 5\}$ receives exactly one solid arrow, so following the arrows backwards defines the inverse $f^{-1}$ (dashed arrows), which returns the shape with that number of corners. In (b) the function $f'$ records only whether a shape has a straight edge. Three shapes share the value yes, so no function undoes $f'$: the value yes does not say which shape produced it*

The best known instance is the inverse of a linear map. A square matrix ${\mathbf A} \in \mathbb{R}^{n \times n}$ represents a linear map of $\mathbb{R}^{n}$ into itself, and that map has an inverse exactly when ${\mathbf A}$ is of full-rank, meaning its columns are linearly independent (Horn and Johnson, 2013, Sect. 0.5). Such a matrix is said to be invertible. The inverse linear map is itself a linear map (Axler, 2024, Thm. 3.63), and the matrix representing it is the inverse matrix ${\mathbf A}^{-1}$, which satisfies

$$
{\mathbf A} {\mathbf A}^{-1} = {\mathbf A}^{-1} {\mathbf A} = \mathbf{I} \text{.}
$$

A square matrix is invertible if and only if its determinant is nonzero (Axler, 2024, Thm. 9.50; Horn and Johnson, 2013, Sect. 0.5). Fig. 2 shows the two linear maps acting on the standard basis of $\mathbb{R}^{2}$.

![Figure 2 of the entry inverse](https://dictionaryofml.org/terms/fig/inverse-fig2.svg)

*Figure 2: Matrix ${\mathbf A}$ representing a linear transformation of $\mathbb{R}^{2}$; inverse matrix ${\mathbf A}^{-1}$ representing the inverse transformation*

For a matrix that is not square or not of full-rank, the one-sided inverses are what remains: a left inverse ${\mathbf B}$ satisfies ${\mathbf B} {\mathbf A} = \mathbf{I}$, and a right inverse ${\mathbf C}$ satisfies ${\mathbf A} {\mathbf C} = \mathbf{I}$. The Moore–Penrose pseudoinverse ${{\mathbf A}}^{+}$ selects one generalized inverse for every matrix (Golub and Loan, 2013, Sect. 5.5.2).

Inverses are used throughout machine learning. Solving a system of linear equations is one task, and ridge regression is another: its closed-form solution inverts the matrix ${\mathbf X}^{\top} {\mathbf X} + \alpha \mathbf{I}$, which is invertible for every positive regularization strength $\alpha$. The sigmoid function and the logit of the opening paragraph are an invertible pair, which allows translating between the score of a linear classifier and a probability value. A feature transformation that is invertible loses no information, since the raw feature vector can be recovered from the transformed one. The inverse of a continuous cumulative distribution function allows generating realizations of a random variable with that cumulative distribution function from a realization of another, simpler random variable, one uniformly distributed on $[0,1]$ (Ross, 2014, Sect. 10.2.1).

A counterfactual explanation for a learned hypothesis can be constructed from a right inverse of it, which returns, for a given prediction, one feature vector of its preimage. A learned hypothesis usually maps many feature vectors to the same prediction, so it has no inverse and no left inverse; a right inverse is what remains, and choosing which element of the preimage it returns is what makes one counterfactual more useful than another.

**See also:** function, injective, preimage, invertible, [matrix](https://dictionaryofml.org/terms/matrix.html), linear map, determinant, pseudoinverse, [linear regression](https://dictionaryofml.org/terms/linreg.html), [sigmoid function](https://dictionaryofml.org/terms/sigmoid.html), logit.

## References

- Horn and Johnson (2013). Matrix Analysis. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781139020411>
- Axler (2024). Linear Algebra Done Right. *Springer*. <https://doi.org/10.1007/978-3-031-41026-0>
- Golub and Loan (2013). Matrix Computations. *The Johns Hopkins Univ. Press*.
- Ross (2014). A First Course in Probability. *Pearson Education*.

## Cite this entry

```bibtex
@misc{dictml_inverse,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {inverse},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/inverse.html}
}
```


---

# kernel method

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/kernelmethod.html) · [typeset PDF](https://dictionaryofml.org/terms/kernelmethod.pdf) · [LaTeX](https://dictionaryofml.org/terms/kernelmethod.tex) · [Python demo](https://dictionaryofml.org/terms/kernelmethod.py) · [notebook](https://dictionaryofml.org/terms/kernelmethod.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A kernel method applies a linear method, such as a linear model or a linear classifier, to transformed feature vectors. The transformation is constructed from a kernel: each feature vector is mapped to a function with domain equal to the original feature space, and inner products between transformed feature vectors reduce to evaluations of a kernel. Kernel methods apply to any feature space on which a kernel is defined, including sets of text documents and graphs.

## Definition

Deciding whether an image shows a specific object (Lampert, 2009), or which topic a text document belongs to (Cristianini and Shawe-Taylor, 2000), are tasks for kernel methods. A kernel method is a machine learning method that applies a linear method, such as a linear model for regression or a linear classifier for classification, to transformed feature vectors that are constructed from a kernel (Lampert, 2009; Schölkopf and Smola, 2002).

Linear methods learn a hypothesis that combines the features of a data point linearly, e.g., $h({\mathbf x}) = {\mathbf w}^{\top} {\mathbf x}$ for the feature space $\mathcal{X} = \mathbb{R}^{d}$. Such a hypothesis predicts well only if the relation between the feature vectors and the labels of data points is approximately linear. Moreover, a linear method requires numeric feature vectors and is not directly applicable to data points from an arbitrary feature space $\mathcal{X}$, such as text documents or graphs.

Both limitations can be mitigated by a feature transformation $\phi: \mathcal{X} \rightarrow \mathcal{H}$ from the original feature space $\mathcal{X}$ to a transformed feature space $\mathcal{H}$, chosen as a Hilbert space. Fig. 1 shows a binary classification problem with the first limitation: no straight line separates the two classes in the original feature space, so no linear classifier has a decision boundary that classifies them correctly. A suitable feature transformation makes a dataset ``more linear'': the relation between the transformed feature vectors $\phi({\mathbf x})$ and the labels is closer to linear in $\mathcal{H}$ than the relation between the original feature vectors ${\mathbf x}$ and the labels is in $\mathcal{X}$.

![Figure 1 of the entry kernelmethod](https://dictionaryofml.org/terms/fig/kernelmethod-fig1.svg)

*Figure 1: Five data points characterized by feature vectors ${\mathbf x}^{(r)}$ and labels $y^{(r)} \in \{ \circ, \square \}$ for $r=1, \,\ldots, \,5$. The curved boundary indicates the original feature space $\mathcal{X}$ (left); the parallelogram indicates a section of the transformed feature space $\mathcal{H}$ (right), which is a linear space (see Hilbert space). With these feature vectors, there is no way to separate the two classes by a straight line (representing the decision boundary of a linear classifier). The dashed lines illustrate three failed attempts: each places the feature vector ${\mathbf x}^{(1)}$ (class $\square$) on the same side as at least one feature vector of class $\circ$, so a linear classifier with any of these decision boundaries misclassifies at least one data point. In contrast, the transformed feature vectors ${\mathbf z}^{(r)} = \phi\big({\mathbf x}^{(r)}\big)$ allow the data points to be separated using a linear classifier*

A useful feature transformation $\phi$ often delivers transformed feature vectors in a Hilbert space $\mathcal{H}$ of high (possibly infinite) dimension (Schölkopf and Smola, 2002). Computing and storing the transformed feature vectors $\phi({\mathbf x})$ explicitly can then be infeasible. However, linear methods access the feature vectors of data points only through inner products between pairs of feature vectors (see linear model). For example, the prediction delivered by a trained support vector machine is a weighted sum of inner products between the feature vector of a new data point and the feature vectors in the training set (Hastie et al., 2009, Eq. (12.20)). To apply a linear method to the transformed feature vectors, it is therefore enough to know the inner products $\langle \phi({\mathbf x}), \phi({\mathbf x}') \rangle$ for every possible pair of feature vectors ${\mathbf x}, {\mathbf x}' \in \mathcal{X}$ (Boser et al., 1992).

These inner products are the values of a single function of two arguments, the kernel defined by

$$
k\big({\mathbf x},{\mathbf x}'\big)
:= \langle \phi({\mathbf x}), \phi({\mathbf x}') \rangle
\text{, for } {\mathbf x}, {\mathbf x}' \in \mathcal{X} \text{.}
$$

The value $k\big({\mathbf x},{\mathbf x}'\big)$ quantifies the similarity of the feature vectors ${\mathbf x}$ and ${\mathbf x}'$. As an inner product of transformed feature vectors, the kernel is symmetric, and every matrix of pairwise kernel values is positive semi-definite; these two properties characterize kernels (Shalev-Shwartz and Ben-David, 2014, Lemma 16.2). A linear method that is applied to the transformed feature vectors thus requires only the kernel $k$; the feature transformation $\phi$ itself is never evaluated.

Kernel methods reverse this construction: they start from a kernel $k: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ and construct the feature transformation from it. The feature transformation sends a feature vector ${\mathbf x} \in \mathcal{X}$ to the function $k\big({\mathbf x},\cdot\big)$, i.e., the transformed feature vector ${\mathbf z} := \phi({\mathbf x}) = k\big({\mathbf x},\cdot\big)$ is itself a function with domain $\mathcal{X}$, for every ${\mathbf x} \in \mathcal{X}$. These functions belong to the reproducing kernel Hilbert space $\mathcal{H}_{k}$ associated with the kernel $k$ (Aronszajn, 1950, Part I, Sect. 2).

Take two original feature vectors ${\mathbf x}, {\mathbf x}' \in \mathcal{X}$. The inner product between their transformed feature vectors is a single kernel evaluation,

$$
\langle k\big({\mathbf x},\cdot\big), k\big({\mathbf x}',\cdot\big) \rangle
= k\big({\mathbf x},{\mathbf x}'\big) \text{.}
$$

More generally, the inner product between the transformed feature vector $k\big({\mathbf x},\cdot\big)$ and any function $h \in \mathcal{H}_{k}$ is a point evaluation,

$$
\langle h, k\big({\mathbf x},\cdot\big) \rangle = h({\mathbf x}) \text{,}
$$

which is referred to as the reproducing property of the reproducing kernel Hilbert space $\mathcal{H}_{k}$ (Aronszajn, 1950, Part I, Sect. 1). This reproducing property contains $\langle k\big({\mathbf x},\cdot\big), k\big({\mathbf x}',\cdot\big) \rangle
= k\big({\mathbf x},{\mathbf x}'\big)$ as a special case that is obtained for the choice $h = k\big({\mathbf x}',\cdot\big)$. So the conclusion of the forward construction holds here too: the (possibly infinite-dimensional) transformed feature vectors never need to be computed explicitly.

Fig. 2 depicts the reproducing kernel Hilbert space $\mathcal{H}_{k}$ for a training set with two data points: a hypothesis $h$ is a single vector in $\mathcal{H}_{k}$, the two transformed feature vectors span a subspace, and $h$ has an orthogonal projection onto it. The value $\langle h, k\big({\mathbf x},\cdot\big) \rangle = h({\mathbf x})$ is a linear function of the transformed feature vector $k\big({\mathbf x},\cdot\big) \in \mathcal{H}_{k}$, but the function ${\mathbf x} \mapsto h({\mathbf x})$ induced on the original feature space $\mathcal{X}$ is nonlinear in general. Linearity in the original feature vector ${\mathbf x}$ may not even be defined, since the feature space $\mathcal{X}$ need not be a vector space, e.g., when it consists of text documents or graphs.

![Figure 2 of the entry kernelmethod](https://dictionaryofml.org/terms/fig/kernelmethod-fig2.svg)

*Figure 2: The reproducing kernel Hilbert space $\mathcal{H}_{k}$ for a training set with two data points having feature vectors ${\mathbf x}^{(1)}, {\mathbf x}^{(2)}$. Their transformed feature vectors $k\big({\mathbf x}^{(1)},\cdot\big)$ and $k\big({\mathbf x}^{(2)},\cdot\big)$ span a two-dimensional subspace (shaded parallelogram). A hypothesis $h$ is a vector in $\mathcal{H}_{k}$; its prediction for a data point with feature vector ${\mathbf x}$ is the inner product $\langle h, k\big({\mathbf x},\cdot\big) \rangle = h({\mathbf x})$. This prediction is a linear function of the transformed feature vector $k\big({\mathbf x},\cdot\big)$, but the function ${\mathbf x} \mapsto h({\mathbf x})$ induced on $\mathcal{X}$ is nonlinear in general. The dashed arrow depicts the orthogonal projection $\widetilde{h}$ of $h$ onto the shaded subspace*

Kernel methods can be formulated as regularized empirical risk minimization over the reproducing kernel Hilbert space $\mathcal{H}_{k}$,

$$
\min_{h \in \mathcal{H}_{k}} \frac{1}{m}
\sum_{r=1}^{m}
L\left(\big({\mathbf x}^{(r)}, y^{(r)}\big),h \right)
+ \alpha \left\lVert h \right\rVert_{\mathcal{H}_{k}}^{2}
\text{,}
$$

over a training set with feature vectors ${\mathbf x}^{(r)}$ and labels $y^{(r)}$, for $r = 1, \ldots, m$. The regularization parameter $\alpha > 0$ weights the penalty term $\left\lVert h \right\rVert_{\mathcal{H}_{k}}^{2}$. The loss function $L\left(\big({\mathbf x}^{(r)}, y^{(r)}\big),h \right)$ depends on $h$ only through the prediction $h\big({\mathbf x}^{(r)}\big)$, i.e., it is a function of the prediction $h\big({\mathbf x}^{(r)}\big)$ and the label $y^{(r)}$ only. Since $h \in \mathcal{H}_{k}$, the reproducing property delivers this prediction as an inner product, $h\big({\mathbf x}^{(r)}\big) = \langle h, k\big({\mathbf x}^{(r)},\cdot\big) \rangle$. By the representer theorem, this optimization problem has a minimizer of the form $\widehat{h} = \sum_{r=1}^{m} \beta_{r} k\big({\mathbf x}^{(r)},\cdot\big)$ with expansion coefficients $\beta_{1}, \ldots, \beta_{m} \in \mathbb{R}$ (Schölkopf and Smola, 2002, Thm. 4.2; Shalev-Shwartz and Ben-David, 2014, Thm. 16.1).

Fig. 2 illustrates the underlying projection argument. Replacing a hypothesis $h$ by its orthogonal projection $\widetilde{h}$ onto the subspace spanned by $k\big({\mathbf x}^{(1)},\cdot\big), \ldots, k\big({\mathbf x}^{(m)},\cdot\big)$ leaves the predictions on the training set unchanged and never increases the penalty term. Some minimizer $\widehat{h}$ therefore lies in this subspace and has the above form.

Training thus reduces to an optimization problem in these $m$ coefficients. For a loss function that is convex in the prediction, it is a convex optimization problem: each prediction $\widehat{h}\big({\mathbf x}^{(r')}\big) = \sum_{r=1}^{m} \beta_{r} k\big({\mathbf x}^{(r)},{\mathbf x}^{(r')}\big)$ on the training set is linear in the coefficients, and the penalty term $\left\lVert \widehat{h} \right\rVert_{\mathcal{H}_{k}}^{2} = \sum_{r=1}^{m} \sum_{r'=1}^{m} \beta_{r} \beta_{r'} k\big({\mathbf x}^{(r)},{\mathbf x}^{(r')}\big)$, obtained from the reproducing property, is a quadratic form with the positive semi-definite matrix of pairwise kernel values and hence convex. The prediction $\widehat{h}({\mathbf x}) = \sum_{r=1}^{m} \beta_{r} k\big({\mathbf x}^{(r)},{\mathbf x}\big)$ requires only kernel evaluations. Examples of such kernel methods (Cristianini and Shawe-Taylor, 2000) are the kernel support vector machine, obtained for the hinge loss (Hastie et al., 2009, Sect. 12.3.2), and kernel ridge regression, obtained for the squared error loss (Hastie et al., 2009, Sect. 5.8.1). For $\mathcal{X} = \mathbb{R}^{d}$, a widely used choice of the kernel $k$ is the Gaussian kernel $k\big({\mathbf x},{\mathbf x}'\big) = \exp\big(- \left\lVert {\mathbf x} - {\mathbf x}' \right\rVert_{2}^{2} / (2 \sigma^{2}) \big)$ with bandwidth $\sigma > 0$. Kernels are also available for data points without numeric features, such as strings and graphs (Schölkopf and Smola, 2002). Fig. 3 shows the nonlinear decision boundary $\widehat{h}({\mathbf x}) = 0$ of a hypothesis $\widehat{h}$ learned by a Gaussian-kernel method with squared error loss from a training set of two concentric rings that no linear classifier separates. Here, kernel ridge regression fits the labels $y^{(r)} \in \{-1, +1\}$ and the prediction is the sign of the fitted value, a construction referred to as regularized least-squares classification (Rifkin et al., 2003).

![Figure 3 of the entry kernelmethod](https://dictionaryofml.org/terms/fig/kernelmethod-fig3.svg)

*Figure 3: A Gaussian-kernel method (kernel ridge regression with $\sigma = 0.8$, $\alpha = 10^{-3}$) trained on a training set of two concentric rings. Data points with label $y = +1$ are drawn as filled circles, those with $y = -1$ as open squares. The decision boundary $\widehat{h}({\mathbf x}) = 0$ of the learned hypothesis (solid closed curve) encircles the inner ring. The dataset is not linearly separable in the original feature space. Data generated by pythondemos/kernelmethod.py*

Kernel methods impose the smoothness assumption through the reproducing kernel Hilbert space norm. That norm $\left\lVert h \right\rVert_{\mathcal{H}_{k}}$ quantifies the smoothness of a hypothesis $h$: it is large for a rapidly varying $h$ and small for a smooth $h$. Moreover, different kernels impose different kinds of smoothness. The Gaussian kernel is infinitely differentiable (Rasmussen and Williams, 2006, Sect. 4.2.1), and so is every $h({\mathbf x}) = \langle h, k\big({\mathbf x},\cdot\big) \rangle$ in its reproducing kernel Hilbert space; the reproducing kernel Hilbert space norm penalizes derivatives of all orders and thereby high-frequency components (Schölkopf and Smola, 2002, Sect. 4.4.2). The Matérn kernels are a family of kernels indexed by a smoothness parameter $\nu > 0$: their reproducing kernel Hilbert space coincides, with equivalent norms, with the Sobolev space of order determined by $\nu$, whose functions have derivatives up to that order in the weak sense (Rasmussen and Williams, 2006, Eq. 4.15; Kanagawa et al., 2018, Example 2.6). The penalty term $\alpha \left\lVert h \right\rVert_{\mathcal{H}_{k}}^{2}$ in the above regularized empirical risk minimization is the squared reproducing kernel Hilbert space norm. It therefore implements smoothness as quantified by $\left\lVert h \right\rVert_{\mathcal{H}_{k}}$: among hypotheses with the same training error, the regularized empirical risk minimization objective function is smallest for the hypothesis with the smallest reproducing kernel Hilbert space norm.

The choice of kernel determines not only training but also the predictions of the learned hypothesis: for localized kernels, such as the Gaussian kernel, the prediction $\widehat{h}({\mathbf x})$ is dominated by the data points in the training set whose feature vectors are close to ${\mathbf x}$. In this sense, kernel methods implement the smoothness assumption: data points with nearby feature vectors obtain similar predictions. The locality of the predictions also makes kernel methods a soft-weighted analogue of k-nearest neighbors: instead of averaging the labels of a fixed number of nearest data points, the prediction weights all data points in the training set by the kernel value $k\big({\mathbf x}^{(r)},{\mathbf x}\big)$ (Hastie et al., 2009, Sect. 6.1).

**Synonyms:** kernel machine.

**See also:** kernel, feature transformation, [Hilbert space](https://dictionaryofml.org/terms/hilbertspace.html), reproducing kernel Hilbert space, [support vector machine](https://dictionaryofml.org/terms/svm.html), [kernel ridge regression](https://dictionaryofml.org/terms/kernelridgeregression.html), smoothness assumption.

## References

- Lampert (2009). Kernel Methods in Computer Vision. *Found. Trends Comput. Graph. Vis.*. <https://doi.org/10.1561/0600000027>
- Cristianini and Shawe-Taylor (2000). An Introduction to Support Vector Machines and Other Kernel-based Learning Methods. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9780511801389>
- Schölkopf and Smola (2002). Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. *MIT Press*. <https://doi.org/10.7551/mitpress/4175.001.0001>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Boser et al. (1992). A Training Algorithm for Optimal Margin Classifiers. *Proc. 5th Annual Workshop on Computational Learning Theory (COLT)*. <https://doi.org/10.1145/130385.130401>
- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Aronszajn (1950). Theory of Reproducing Kernels. *Trans. Am. Math. Soc.*. <https://doi.org/10.1090/S0002-9947-1950-0051437-7>
- Rifkin et al. (2003). Regularized Least-Squares Classification. *Advances in Learning Theory: Methods, Model and Applications*. <https://doi.org/10.21236/ada454981>
- Rasmussen and Williams (2006). Gaussian Processes for Machine Learning. *MIT Press*. <https://doi.org/10.7551/mitpress/3206.001.0001>
- Kanagawa et al. (2018). Gaussian Processes and Kernel Methods: A Review on Connections and Equivalences. <https://arxiv.org/abs/1807.02582>

## Cite this entry

```bibtex
@incollection{dictml_kernelmethod,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {kernel method},
  booktitle = {Dictionary of Applied Machine Learning},
  publisher = {Springer Nature Singapore},
  year = {2026},
  doi = {10.1007/978-981-92-2012-0_16-1},
  isbn = {978-981-92-2012-0},
  url = {https://doi.org/10.1007/978-981-92-2012-0_16-1}
}
```


---

# kernel ridge regression

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-11. [HTML](https://dictionaryofml.org/terms/kernelridgeregression.html) · [typeset PDF](https://dictionaryofml.org/terms/kernelridgeregression.pdf) · [LaTeX](https://dictionaryofml.org/terms/kernelridgeregression.tex) · [Python demo](https://dictionaryofml.org/terms/kernelridgeregression.py) · [notebook](https://dictionaryofml.org/terms/kernelridgeregression.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Kernel ridge regression is a kernel method for regression that applies ridge regression to transformed feature vectors constructed from a kernel, and thereby learns a nonlinear hypothesis in closed form. It performs regularized empirical risk minimization over the reproducing kernel Hilbert space of the kernel, combining the squared error loss with a squared-norm penalty term. By the representer theorem, the learned hypothesis is a weighted sum of kernel evaluations at the feature vectors in the training set, and the coefficients are obtained in closed form by solving a single system of linear equations. Like ridge regression, the method equals empirical risk minimization on an augmented training set: the transformed feature vectors are perturbed by realizations of a zero-mean Gaussian process whose covariance function is the kernel scaled by the regularization parameter. For the kernel that is a modified inner product of the feature space, kernel ridge regression is ridge regression with a matching penalty term. The perturbed copies of a data point then fill an ellipse whose shape the kernel sets, the circle of ridge regression for the standard inner product.

## Definition

The temperature over the course of a day rises and falls, so predicting it from the time of the day is a regression problem whose relation between feature and label is nonlinear. Kernel ridge regression solves such regression problems by applying ridge regression to transformed feature vectors constructed from a kernel, and thereby learns a nonlinear hypothesis in closed form (Hastie et al., 2009, Sect. 5.8.2; Schölkopf and Smola, 2002). Fig. 1 shows a training set with a periodic relation between feature and label. With the Gaussian kernel $k\big(x,x'\big) = \exp\big( - (x - x')^{2} / (2 \sigma^{2}) \big)$ of bandwidth $\sigma = 0.7$, the learned hypothesis follows the data points smoothly, with a training mean squared error of $0.013$, while ridge regression on the raw feature can only learn a linear hypothesis, with a training mean squared error of $0.36$.

Formally, kernel ridge regression learns a hypothesis from the reproducing kernel Hilbert space $\mathcal{H}_{k}$ of a kernel $k: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ by minimizing the penalized average squared error loss on a training set $\mathcal{D}^{(\rm train)} = \big\{ \left( {\mathbf x}^{(r)},y^{(r)} \right) \big\}_{r=1}^{m}$ with numeric labels $y^{(r)} \in \mathbb{R}$,

\begin{equation}
\tag{1}
\widehat{h} = \mathop{\mathrm{arg\,min}}_{h \in \mathcal{H}_{k}}
\frac{1}{m} \sum_{r=1}^{m}
\big( y^{(r)} - h\big({\mathbf x}^{(r)}\big) \big)^{2}
+ \alpha \left\lVert h \right\rVert_{\mathcal{H}_{k}}^{2}
\text{.}
\end{equation}

The penalty term is the scaled squared reproducing kernel Hilbert space norm $\alpha \left\lVert h \right\rVert_{\mathcal{H}_{k}}^{2}$ with a regularization parameter $\alpha > 0$. The resulting method is an instance of regularized empirical risk minimization, as is ridge regression, whose penalty term is the scaled squared Euclidean norm $\alpha \left\lVert {\mathbf w} \right\rVert_{2}^{2}$ of the model parameters (Hastie et al., 2009, Sect. 5.8.2). The average squared error loss is a convex function of $h$, and the squared norm is strictly convex. The kernel ridge regression problem Eq. (1) therefore has a unique minimizer $\widehat{h}$ for any training set, as does ridge regression.

By the representer theorem (see kernel method) (Schölkopf and Smola, 2002, Thm. 4.2), the minimizer of the kernel ridge regression problem Eq. (1) has the form $\widehat{h} = \sum_{r=1}^{m} \beta_{r} k\big({\mathbf x}^{(r)},\cdot\big)$ with a coefficient vector $\boldsymbol{\beta} = \big(\beta_{1}, \ldots, \beta_{m}\big)^{\top}$. Inserting this form into the kernel ridge regression problem Eq. (1) yields the finite-dimensional optimization problem

$$
\min_{\boldsymbol{\beta} \in \mathbb{R}^{m}}
\frac{1}{m}
\left\lVert {\mathbf y} - \mathbf{K} \boldsymbol{\beta} \right\rVert_{2}^{2}
+ \alpha\, \boldsymbol{\beta}^{\top} \mathbf{K} \boldsymbol{\beta}
\text{,}
$$

with the label vector ${\mathbf y} = \big(y^{(1)}, \ldots, y^{(m)}\big)^{\top}$ and the Gram matrix $\mathbf{K}$ with entries $K_{r,r'} = k\big({\mathbf x}^{(r)},{\mathbf x}^{(r')}\big)$ (see kernel).

By the zero-gradient condition, a coefficient vector solving this optimization problem satisfies the system of linear equations

$$
\mathbf{K}\big( \big(\mathbf{K} + \alpha m \mathbf{I}\big) \boldsymbol{\beta} - {\mathbf y} \big) = \mathbf{0} \text{.}
$$

Since $\mathbf{K}$ is positive semi-definite and $\alpha > 0$, the matrix $\mathbf{K} + \alpha m \mathbf{I}$ is invertible, and a minimizer is obtained in closed form as

\begin{equation}
\tag{2}
\widehat{\boldsymbol{\beta}}
= \big( \mathbf{K} + \alpha m \mathbf{I} \big)^{-1} {\mathbf y}
\text{.}
\end{equation}

The prediction for a data point with feature vector ${\mathbf x}$ is

$$
\widehat{h}({\mathbf x}) = \sum_{r=1}^{m} \widehat{\beta}_{r} k\big({\mathbf x}^{(r)},{\mathbf x}\big) \text{.}
$$

Training is a one-shot computation: kernel ridge regression requires no iterative optimization method (and hence no fixed-point interpretation), only the unique solution of the $m \times m$ system of linear equations with the invertible matrix $\mathbf{K} + \alpha m \mathbf{I}$, which is the closed form Eq. (2).

![Figure 1 of the entry kernelridgeregression](https://dictionaryofml.org/terms/fig/kernelridgeregression-fig1.svg)

*Figure 1: Kernel ridge regression with a Gaussian kernel ($\sigma = 0.7$, $\alpha = 10^{-2}$) on a training set of $m = 20$ noisy data points (circles) generated from a periodic function. The learned hypothesis $\widehat{h}$ (solid curve) follows the nonlinear relation; ridge regression on the raw feature (dashed line) can only learn a linear hypothesis. Its training mean squared error is more than ten times larger than that of kernel ridge regression. Data generated by pythondemos/kernelridgeregression.py*

For the linear kernel $k\big({\mathbf x},{\mathbf x}'\big) = {\mathbf x}^{\top} {\mathbf x}'$ on $\mathcal{X} = \mathbb{R}^{d}$, kernel ridge regression produces the same predictions as ridge regression applied to the raw feature vectors (Rasmussen and Williams, 2006, Sect. 2.1, Eqs. (2.11)–(2.12)). Indeed, the Gram matrix is then $\mathbf{K} = {\mathbf X} {\mathbf X}^{\top}$, with the feature matrix ${\mathbf X} = \big({\mathbf x}^{(1)}, \ldots, {\mathbf x}^{(m)}\big)^{\top}$. Multiplying out both sides shows that $\big({\mathbf X}^{\top} {\mathbf X} + \alpha m \mathbf{I}\big) {\mathbf X}^{\top}
= {\mathbf X}^{\top} \big({\mathbf X} {\mathbf X}^{\top} + \alpha m \mathbf{I}\big)$, and hence ${\mathbf X}^{\top} \big(\mathbf{K} + \alpha m \mathbf{I}\big)^{-1}
= \big({\mathbf X}^{\top} {\mathbf X} + \alpha m \mathbf{I}\big)^{-1} {\mathbf X}^{\top}$. The prediction $\widehat{h}({\mathbf x}) = {\mathbf x}^{\top} {\mathbf X}^{\top} \widehat{\boldsymbol{\beta}}$ of kernel ridge regression therefore equals ${\mathbf x}^{\top} \big({\mathbf X}^{\top} {\mathbf X} + \alpha m \mathbf{I}\big)^{-1} {\mathbf X}^{\top} {\mathbf y}$, the prediction of ridge regression.

Like ridge regression, kernel ridge regression has an interpretation as a form of data augmentation. Training with noise added to the feature vectors is equivalent to adding a penalty term (Bishop, 1995, Eqs. (10)–(11)). Replacing each data point by a probability distribution centered at it is the vicinal form of empirical risk minimization (Chapelle et al., 2001), and its effect on kernel methods is a regularization of the learned hypothesis (Dao et al., 2019).

The following example makes the equivalence explicit for a linear kernel on the plane, so that the perturbed copies can be drawn. Consider data points with two features, the feature space $\mathcal{X} = \mathbb{R}^{2}$, and the kernel

\begin{equation}
\tag{3}
k\big({\mathbf x},{\mathbf x}'\big) := {\mathbf x}^{\top} {\mathbf C}^{-1} {\mathbf x}' \text{,}
\end{equation}

with a positive definite matrix ${\mathbf C} \in \mathbb{R}^{2 \times 2}$. This kernel is an inner product of $\mathbb{R}^{2}$ with a modified metric: the squared norm $k\big({\mathbf x},{\mathbf x}\big) = {\mathbf x}^{\top} {\mathbf C}^{-1} {\mathbf x}$ scales the two features differently and couples them, and its unit ball is an ellipse instead of the circle of the Euclidean norm. The reproducing kernel Hilbert space $\mathcal{H}_{k}$ of this kernel is $\mathbb{R}^{2}$ with this inner product. The transformed feature vector $k\big({\mathbf x},\cdot\big)$ is the linear function ${\mathbf x}' \mapsto {\mathbf x}^{\top} {\mathbf C}^{-1} {\mathbf x}'$, every hypothesis in $\mathcal{H}_{k}$ is a linear model $h^{({\mathbf w})}({\mathbf x}) = {\mathbf w}^{\top} {\mathbf x}$ with model parameters ${\mathbf w} \in \mathbb{R}^{2}$, and its reproducing kernel Hilbert space norm is $\left\lVert h^{({\mathbf w})} \right\rVert_{\mathcal{H}_{k}}^{2} = {\mathbf w}^{\top} {\mathbf C} {\mathbf w}$.

The kernel ridge regression problem Eq. (1) therefore reads

\begin{equation}
\tag{4}
\widehat{{\mathbf w}} = \mathop{\mathrm{arg\,min}}_{{\mathbf w} \in \mathbb{R}^{2}}
\frac{1}{m} \sum_{r=1}^{m}
\big( y^{(r)} - {\mathbf w}^{\top} {\mathbf x}^{(r)} \big)^{2}
+ \alpha \, {\mathbf w}^{\top} {\mathbf C} {\mathbf w} \text{,}
\end{equation}

which is ridge regression with the penalty term $\alpha \left\lVert {\mathbf w} \right\rVert_{2}^{2}$ replaced by $\alpha \, {\mathbf w}^{\top} {\mathbf C} {\mathbf w}$. Its unique minimizer is $\widehat{{\mathbf w}} = \big( {\mathbf X}^{\top} {\mathbf X} + \alpha m {\mathbf C} \big)^{-1} {\mathbf X}^{\top} {\mathbf y}$ with the feature matrix ${\mathbf X} = \big({\mathbf x}^{(1)}, \ldots, {\mathbf x}^{(m)}\big)^{\top}$, and it coincides with ${\mathbf C}^{-1} {\mathbf X}^{\top} \widehat{\boldsymbol{\beta}}$ for the coefficients Eq. (2) with the Gram matrix $\mathbf{K} = {\mathbf X} {\mathbf C}^{-1} {\mathbf X}^{\top}$. For ${\mathbf C} = \mathbf{I}$, the kernel Eq. (3) is the linear kernel and the penalized problem Eq. (4) is ridge regression itself.

Adding the penalty term in the penalized problem Eq. (4) is equivalent to empirical risk minimization on an augmented training set, i.e., to computing the average squared error loss of $h^{({\mathbf w})}$ on it. This augmented training set is obtained from the original training set. For each data point $\left( {\mathbf x}^{(r)},y^{(r)} \right)$, $r = 1, \ldots, m$, the feature vector ${\mathbf x}^{(r)}$ is replaced by realizations of infinitely many independent and identically distributed RVs ${\mathbf x}^{(r)} + {\boldsymbol \varepsilon}^{(r)}$ with Gaussian RVs ${\boldsymbol \varepsilon}^{(r)} \sim \mathcal{N}\left(\mathbf{0},\alpha {\mathbf C}\right)$. The label $y^{(r)}$ is left unchanged.

The penalty term is exactly the amount by which the average squared error loss of $h^{({\mathbf w})}$ over the perturbed copies exceeds its squared error loss on the original data point. Since ${\mathbb{E}} \{ {\boldsymbol \varepsilon}^{(r)} \} = \mathbf{0}$, the increase is

$$
{\mathbb{E}} \big\{ \big( {\mathbf w}^{\top} {\boldsymbol \varepsilon}^{(r)} \big)^{2} \big\}
= {\mathbf w}^{\top} {\mathbb{E}} \big\{ {\boldsymbol \varepsilon}^{(r)} \big({\boldsymbol \varepsilon}^{(r)}\big)^{\top} \big\} {\mathbf w}
= \alpha \, {\mathbf w}^{\top} {\mathbf C} {\mathbf w} \text{.}
$$

The equivalence is exact for the linear model $h^{({\mathbf w})}$, while for a nonlinear hypothesis it holds up to terms of higher order in the noise amplitude (Bishop, 1995, Eqs. (10)–(11)). As in ridge regression, this increase of the average loss under data augmentation is an estimate for the generalization gap (Hastie et al., 2009, Sect. 7.4). Accordingly, linear regression on a large augmented training set recovers the predictions of kernel ridge regression.

The kernel determines the shape of these perturbations (see Fig. 2). The covariance matrix $\alpha {\mathbf C}$ of ${\boldsymbol \varepsilon}^{(r)}$ is the scaled inverse of the matrix in the kernel Eq. (3). The one-standard-deviation contour of the perturbed copies is therefore

$$
\big({\mathbf x} - {\mathbf x}^{(r)}\big)^{\top} {\mathbf C}^{-1} \big({\mathbf x} - {\mathbf x}^{(r)}\big) = \alpha \text{.}
$$

This contour is the set of feature vectors ${\mathbf x}$ with $k\big({\mathbf x} - {\mathbf x}^{(r)},{\mathbf x} - {\mathbf x}^{(r)}\big) = \alpha$, i.e., the ball of radius $\sqrt{\alpha}$ around ${\mathbf x}^{(r)}$ in the norm induced by the kernel. For ${\mathbf C} = \mathbf{I}$, this ball is a circle of radius $\sqrt{\alpha}$: the perturbations in the data augmentation interpretation of ridge regression have covariance matrix $\alpha \mathbf{I}$ and no preferred direction. For a general ${\mathbf C}$ it is an ellipse, so the ellipses drawn around the data points display the kernel. The size of a perturbation is measured by its kernel norm, and the penalty term $\alpha \, {\mathbf w}^{\top} {\mathbf C} {\mathbf w}$ is the average squared error loss that perturbations of kernel norm $\sqrt{\alpha}$ cause.

The ellipses also display the matrix ${\mathbf C}$ itself. Its eigenvalue decomposition ${\mathbf C} = \sum_{j=1}^{2} \lambda_{j} \mathbf{u}^{(j)} \big(\mathbf{u}^{(j)}\big)^{\top}$, with eigenvalues $\lambda_{1} \geq \lambda_{2} > 0$ and orthonormal eigenvectors $\mathbf{u}^{(1)}, \mathbf{u}^{(2)}$, turns the contour into

$$
\sum_{j=1}^{2} \frac{\big( \big(\mathbf{u}^{(j)}\big)^{\top} \big({\mathbf x} - {\mathbf x}^{(r)}\big) \big)^{2}}{\alpha \lambda_{j}} = 1 \text{.}
$$

The principal axes of every ellipse therefore point along the eigenvectors of ${\mathbf C}$, and the semi-axes have lengths $\sqrt{\alpha \lambda_{1}}$ and $\sqrt{\alpha \lambda_{2}}$. The perturbations spread most along the eigenvector with the largest eigenvalue, and the penalty term grows with the square of a displacement of ${\mathbf w}$ along $\mathbf{u}^{(j)}$, in proportion to $\lambda_{j}$.

![Figure 2 of the entry kernelridgeregression](https://dictionaryofml.org/terms/fig/kernelridgeregression-fig2.svg)

*Figure 2: The data augmentation interpretation of kernel ridge regression with the kernel Eq. (3), for a training set of six data points with two features, $\alpha = 1/2$, and a matrix ${\mathbf C}$ with eigenvalues $\lambda_{1} = 1.5^{2}$, $\lambda_{2} = 0.75^{2}$ and eigenvectors rotated by $30^{\circ}$ against the feature axes. Each original data point (filled circle) is replaced by three copies (open squares) whose feature vectors are perturbed by Gaussian RVs with covariance matrix $\alpha {\mathbf C}$, keeping the label. The ellipse around each data point is the one-standard-deviation contour of its copies, the ball of radius $\sqrt{\alpha}$ in the norm induced by the kernel; for ${\mathbf C} = \mathbf{I}$ it is the circle of radius $\sqrt{\alpha}$ of the isotropic perturbations of ridge regression. The arrows on one ellipse are its principal axes $\sqrt{\alpha \lambda_{j}}\, \mathbf{u}^{(j)}$, the eigenvectors of ${\mathbf C}$ scaled by the square roots of $\alpha$ times its eigenvalues. The dashed lines are contour lines of the hypothesis $\widehat{h}({\mathbf x}) = \widehat{{\mathbf w}}^{\top} {\mathbf x}$ learned from the six data points by the penalized problem Eq. (4), labeled with its value: a copy displaced along a contour line keeps its prediction, and a copy displaced across the contour lines changes it in proportion to the displacement. Data generated by pythondemos/kernelridgeregression.py*

For the kernel Eq. (3), the perturbation acts linearly on the transformed feature vectors, $k\big({\mathbf x}^{(r)} + {\boldsymbol \varepsilon}^{(r)},\cdot\big) = k\big({\mathbf x}^{(r)},\cdot\big) + k\big({\boldsymbol \varepsilon}^{(r)},\cdot\big)$. The random function $k\big({\boldsymbol \varepsilon}^{(r)},\cdot\big)$ is therefore a zero-mean Gaussian process whose covariance function is the kernel scaled by the regularization parameter,

\begin{equation}
\tag{5}
{\mathbb{E}} \big\{ k\big({\boldsymbol \varepsilon}^{(r)},{\mathbf x}\big) \, k\big({\boldsymbol \varepsilon}^{(r)},{\mathbf x}'\big) \big\}
= {\mathbf x}^{\top} {\mathbf C}^{-1} {\mathbb{E}} \big\{ {\boldsymbol \varepsilon}^{(r)} \big({\boldsymbol \varepsilon}^{(r)}\big)^{\top} \big\} {\mathbf C}^{-1} {\mathbf x}'
= \alpha \, k\big({\mathbf x},{\mathbf x}'\big) \text{.}
\end{equation}

For a general kernel, the raw feature vectors cannot be perturbed in this way, but the perturbation of the transformed feature vector $k\big({\mathbf x}^{(r)},\cdot\big)$ by a zero-mean Gaussian process with covariance function $\alpha k$ remains defined (Kanagawa et al., 2018, Sect. 4). Up to the scaling by $\alpha$ in the covariance function Eq. (5), this is the Gaussian process with covariance function $k$ and noise variance $\alpha m$ whose posterior mean kernel ridge regression computes: the prediction $\widehat{h}({\mathbf x})$ coincides with that mean (Kanagawa et al., 2018, Prop. 3.6).

**See also:** [kernel method](https://dictionaryofml.org/terms/kernelmethod.html), ridge regression, kernel, reproducing kernel Hilbert space, regularized empirical risk minimization, squared error loss, regression, Gaussian process, [data augmentation](https://dictionaryofml.org/terms/dataaug.html), feature transformation, penalty term.

## References

- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Schölkopf and Smola (2002). Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. *MIT Press*. <https://doi.org/10.7551/mitpress/4175.001.0001>
- Rasmussen and Williams (2006). Gaussian Processes for Machine Learning. *MIT Press*. <https://doi.org/10.7551/mitpress/3206.001.0001>
- Bishop (1995). Training with Noise is Equivalent to Tikhonov Regularization. *Neural Computation*. <https://doi.org/10.1162/neco.1995.7.1.108>
- Chapelle et al. (2001). Vicinal Risk Minimization. *Advances in Neural Information Processing Systems 13 (NIPS 2000)*.
- Dao et al. (2019). A Kernel Theory of Modern Data Augmentation. *Proceedings of the 36th International Conference on Machine Learning (ICML)*.
- Kanagawa et al. (2018). Gaussian Processes and Kernel Methods: A Review on Connections and Equivalences. <https://arxiv.org/abs/1807.02582>

## Cite this entry

```bibtex
@misc{dictml_kernelridgeregression,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {kernel ridge regression},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/kernelridgeregression.html}
}
```


---

# k-fold cross-validation (k-fold CV)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/kfoldcv.html) · [typeset PDF](https://dictionaryofml.org/terms/kfoldcv.pdf) · [LaTeX](https://dictionaryofml.org/terms/kfoldcv.tex) · [Python demo](https://dictionaryofml.org/terms/kfoldcv.py) · [notebook](https://dictionaryofml.org/terms/kfoldcv.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

$k$-fold cross-validation ($k$-fold CV) divides a dataset into $k$ folds of equal size and uses each fold once as the validation set while the other $k-1$ folds form the training set of a given machine learning method. The $k$ resulting validation errors are averaged into a single number, an indicator of generalization beyond the training set. While a single validation step estimates the risk of a single hypothesis, $k$-fold CV estimates the risk of a training method. Leave-one-out cross-validation is the extreme case with one data point per fold, and stratified k-fold cross-validation preserves class proportions of the full dataset in every fold.

## Definition

Four days at Krems an der Donau are the only recorded data points. A day is characterized by its morning minimum temperature as the feature $x$ and its maximum daytime temperature as the label $y$ (Fig. 1). Three of the days form the training set of a polynomial regression of degree three, and the remaining day is the validation set. A cubic has four coefficients and the training set holds three days, so the empirical risk minimization problem is underdetermined and its solution passes exactly through those three days: the training error is zero whichever day is held back.

That makes the held-back day the only evidence about the learned hypothesis, and a single split can give a small validation error to a hypothesis with a large risk. Holding back the first day gives a validation error of $1.37$, which suggests a reliable hypothesis. This changes when another day is held back for the validation set, as panels (b) to (d) of Fig. 1 illustrate. Using different splits of a dataset into training set and validation set offers a more reliable assessment of a machine learning method.

![Figure 1 of the entry kfoldcv](https://dictionaryofml.org/terms/fig/kfoldcv-fig1.svg)

*Figure 1: (a) Four days at Krems an der Donau, each a data point with the morning minimum temperature as feature $x$ and the maximum daytime temperature as label $y$, and the degree-three hypothesis that empirical risk minimization delivers from three of them; the red open square is the first day, held back as the validation set (fold $b = 1$). (b) to (d) The hypothesis obtained when the second, third or fourth day is held back instead. Data generated by pythondemos/kfoldcv.py*

Consider a machine learning method $\mathcal{A}$ that maps a training set to a learned hypothesis $\widehat{h} =
\mathcal{A}(\mathcal{D}^{(\rm train)})$. $k$-fold CV estimates the expected loss (or risk) of the method $\mathcal{A}$ as follows (Hastie et al., 2009, Sect. 7.10; Stone, 1974). It divides a dataset $\mathcal{D}$ evenly into $k$ subsets, the folds $\mathcal{D}^{(1)},\,\ldots,\,
\mathcal{D}^{(k)}$ (Fig. 2). For each fold $b = 1, \,\ldots, \,k$, the model is trained on the union of all folds except $\mathcal{D}^{(b)}$ and the resulting hypothesis is evaluated on the held-out fold $\mathcal{D}^{(b)}$. The $k$ per-fold validation errors are then averaged to obtain the CV error $E_{\rm CV}$, the CV estimate of the expected loss. Algorithm 1 summarizes this procedure.

![Algorithm 1 of the entry kfoldcv](https://dictionaryofml.org/terms/fig/kfoldcv-alg1.svg)

*Algorithm 1: $k$-fold CV*

A single validation set assesses one hypothesis, e.g., the one that $\mathcal{A}$ delivers from a training set (Bates et al., 2023, Sect. 3.3.1). In contrast, $k$-fold CV has no single hypothesis to assess, since every iteration learns a potentially different hypothesis. Thus, the average validation error $E_{\rm CV}$ that $k$-fold CV delivers measures the method $\mathcal{A}$ rather than one hypothesis: the average risk of the hypotheses obtained from training sets holding the fraction $(k-1)/k$ of the dataset (Bates et al., 2023, Sect. 3; Hastie et al., 2009, Sect. 7.10.1). Compared to a single split into training set and validation set, $k$-fold CV uses each data point for both training and validation. In Fig. 2, for example, each data point has been used four times for training and once for validation after all five iterations. A bootstrap estimate is an average of the same kind (Bates et al., 2023, Sect. 3.5).

![Figure 2 of the entry kfoldcv](https://dictionaryofml.org/terms/fig/kfoldcv-fig2.svg)

*Figure 2: An instance of $k$-fold CV with $k=5$: the available dataset $\mathcal{D}$ is evenly divided into five folds $\mathcal{D}^{(1)},\,\ldots,\,\mathcal{D}^{(5)}$. Each row corresponds to one iteration; the shaded cell marks the fold used as validation set, while the remaining $k-1$ folds form the training set*

Some variants and special cases of $k$-fold CV are:

- *Stratified k-fold cross-validation* preserves the class proportions of the full dataset in every fold, important for classification with imbalanced data (Kohavi, 1995).
- *Leave-one-out cross-validation* is the extreme case $k = m$, with $m$ the number of data points in the dataset, where each fold contains a single data point (Stone, 1974).
- *Repeated $k$-fold CV* averages the results of several runs with different random fold assignments, which reduces the variance contributed by the partitioning (Kim, 2009).

**See also:** validation, [validation error](https://dictionaryofml.org/terms/valerr.html), generalization gap, learning curve, leave-one-out cross-validation, stratified $k$-fold cross-validation, [model selection](https://dictionaryofml.org/terms/modelsel.html), [bootstrap](https://dictionaryofml.org/terms/bootstrap.html).

## References

- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Stone (1974). Cross-Validatory Choice and Assessment of Statistical Predictions. *Journal of the Royal Statistical Society. Series B (Methodological)*. <https://doi.org/10.1111/j.2517-6161.1974.tb00994.x>
- Bates et al. (2023). Cross-Validation: What Does It Estimate and How Well Does It Do It?. *Journal of the American Statistical Association*. <https://doi.org/10.1080/01621459.2023.2197686>
- Kohavi (1995). A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection. *Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI)*.
- Kim (2009). Estimating Classification Error Rate: Repeated Cross-Validation, Repeated Hold-Out and Bootstrap. *Computational Statistics & Data Analysis*.

## Cite this entry

```bibtex
@misc{dictml_kfoldcv,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {$k$-fold cross-validation ($k$-fold CV)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/kfoldcv.html}
}
```


---

# k-means

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-10. [HTML](https://dictionaryofml.org/terms/kmeans.html) · [typeset PDF](https://dictionaryofml.org/terms/kmeans.pdf) · [LaTeX](https://dictionaryofml.org/terms/kmeans.tex) · [Python demo](https://dictionaryofml.org/terms/kmeans.py) · [notebook](https://dictionaryofml.org/terms/kmeans.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

$k$-means is a hard clustering method for data points with numeric feature vectors. Each data point is assigned to exactly one of $k$ clusters, each represented by a cluster centroid, namely the cluster whose cluster centroid is nearest in Euclidean distance. The cluster centroids are chosen to minimize the clustering error, the average squared Euclidean distance between a data point and its nearest cluster centroid. The resulting $k$-means optimization problem is an instance of empirical risk minimization using a specific choice for the loss function. This optimization problem is non-convex and NP-hard, and typically solved using Lloyd's algorithm. This algorithm consists of a fixed-point iteration that never increases the clustering error and reaches a fixed point after finitely many iterations. Replacing the squared Euclidean distance by the Euclidean distance itself gives $k$-medians, whose averaging step computes a geometric median and which is less sensitive to outliers.

## Definition

A photograph of a mountain is cut into small square patches, and each patch is described by two numbers: how green and how blue it is on average (Fig. 1). Vegetation, mountain and haze, and bright sky form three clusters of such patches, which are data points characterized by a feature vector ${\mathbf x} \in \mathbb{R}^{d}$. $k$-means is a hard clustering method for such data points (Hastie et al., 2009, Sect. 14.3.6; MacQueen, 1967).

The idea of $k$-means is to assign each data point of a dataset $\mathcal{D} = \{{\mathbf x}^{(1)}, \ldots,
{\mathbf x}^{(m)}\}$ to exactly one of $k$ clusters indexed by $c = 1, \ldots, k$. Each cluster is represented by a cluster centroid ${{\mathbf w}}^{(c)} \in \mathbb{R}^{d}$, and a data point is assigned to the cluster whose cluster centroid is nearest in Euclidean distance. The quality of a given choice of cluster centroids, and of the resulting hard clustering, is measured by the clustering error: the average squared Euclidean distance between a data point and its nearest cluster centroid,

$$
\frac{1}{m}\sum_{r=1}^{m}
\min_{c = 1, \ldots, k}
\left\lVert {\mathbf x}^{(r)} -
{{\mathbf w}}^{(c)} \right\rVert_{2}^{2}\text{.}
$$

The $k$-means principle is to choose cluster centroids that minimize the clustering error,

$$
\min_{{{\mathbf w}}^{(1)}, \ldots, {{\mathbf w}}^{(k)}}
\;\frac{1}{m}\sum_{r=1}^{m}
\min_{c = 1, \ldots, k}
\left\lVert {\mathbf x}^{(r)} -
{{\mathbf w}}^{(c)} \right\rVert_{2}^{2}\text{.}
$$

The inner minimization assigns each data point to its nearest cluster centroid, and the outer minimization moves the cluster centroids (Fig. 1).

![Figure 1 of the entry kmeans](https://dictionaryofml.org/terms/fig/kmeans-fig1.svg)

*Figure 1: Top: the photograph of the Oetscher massif with the patch grid drawn on it; each patch is one data point. Bottom: the $972$ patches, each described by its average greenness and its average blueness, both on a scale from $0$ to $1$. The three crosses are the cluster centroids found by $k$-means with $k = 3$, labeled at their height, and the dashed lines are the boundaries between the clusters: each patch is assigned to the cluster whose cluster centroid is nearest. The thin lines connect two cluster centroids; each boundary crosses such a line at its midpoint at a right angle, marked by the small square. Data generated by pythondemos/kmeans.py*

The $k$-means optimization problem has no closed-form solution, and solving it exactly is NP-hard (Mahajan et al., 2009). The standard method for solving it approximately is Lloyd's algorithm (Lloyd, 1982), which alternates between two steps. The assignment step assigns every data point to its nearest cluster centroid,

\begin{equation}
\tag{1}
\widehat{c}^{(r)} :=
\mathop{\mathrm{arg\,min}}_{c = 1, \ldots, k}
\left\lVert {\mathbf x}^{(r)} -
{{\mathbf w}}^{(c)} \right\rVert_{2}
\quad \text{for } r = 1, \ldots, m\text{,}
\end{equation}

and the averaging step moves each cluster centroid to the mean of the data points assigned to it,

\begin{equation}
\tag{2}
{{\mathbf w}}^{(c)} :=
\frac{1}{\big|\{r : \widehat{c}^{(r)} = c\}\big|}
\sum_{r : \widehat{c}^{(r)} = c}
{\mathbf x}^{(r)}
\quad \text{for } c = 1, \ldots, k\text{.}
\end{equation}

The two steps are repeated until the assignment Eq. (1) stops changing. Algorithm 1 states the method in full.

![Algorithm 1 of the entry kmeans](https://dictionaryofml.org/terms/fig/kmeans-alg1.svg)

*Algorithm 1: $k$-means, Lloyd's algorithm*

Run on the $972$ patches of Fig. 1 with $k = 3$, $k$-means separates vegetation, mountain and haze, and bright sky. Every patch it assigns to the bluest cluster centroid lies in the upper half of the photograph. The learned cluster centroids define a hard clustering: every data point belongs to exactly one cluster, the one with the nearest cluster centroid. Fig. 2 shows this assignment on the photograph itself.

![Figure 2 of the entry kmeans](https://dictionaryofml.org/terms/fig/kmeans-fig2.svg)

*Figure 2: (a) The photograph from Fig. 1 at patch resolution, each patch one data point. The same photograph three times, each keeping only the patches assigned to one cluster: vegetation (b), mountain and haze (c), and bright sky (d). Every patch is bright in exactly one copy and black in the other two, so the three copies add back up to (a). Data generated by pythondemos/kmeans.py*

The $k$-means principle is an instance of empirical risk minimization with a particular choice for the hypothesis space and the loss function. The hypothesis space $\mathcal{H}_{k}$ consists of all maps $h: \mathbb{R}^{d} \rightarrow \mathbb{R}^{d}$ of the piecewise-constant form

$$
h({\mathbf x}) =
{{\mathbf w}}^{(\,c^{\star}({\mathbf x}))},
\qquad
c^{\star}({\mathbf x}) =
\mathop{\mathrm{arg\,min}}_{c = 1, \ldots, k}
\left\lVert {\mathbf x} -
{{\mathbf w}}^{(c)} \right\rVert_{2}\text{,}
$$

parameterized by the $k$ cluster centroids ${{\mathbf w}}^{(1)}, \ldots, {{\mathbf w}}^{(k)}
\in \mathbb{R}^{d}$. Each such hypothesis maps every feature vector to its nearest cluster centroid. The region of feature space mapped to one cluster centroid is bounded by hyperplanes, one for each other cluster centroid (the dashed lines of Fig. 1). Thus, every cluster found by $k$-means is a convex region. The loss incurred by a feature vector ${\mathbf x}$ is the squared Euclidean distance to its nearest cluster centroid, $L\left({\mathbf x},h \right) = \left\lVert {\mathbf x} - h({\mathbf x}) \right\rVert_{2}^{2}$. The clustering error is then obtained as the average loss, i.e., the empirical risk, over the entire dataset. Minimizing the clustering error is therefore empirical risk minimization over $\mathcal{H}_{k}$.

The clustering error is a non-convex function of the cluster centroids: with a single data point at $x = 0$ and two scalar cluster centroids $w^{(1)}, w^{(2)} \in \mathbb{R}$, it is $\min\big((w^{(1)})^{2}, (w^{(2)})^{2}\big)$, so both $(1, 0)$ and $(0, 1)$ are global optima with value $0$, but their midpoint $(0.5, 0.5)$ gives $0.25$. Since Lloyd's algorithm might converge to a local minimum of the clustering error, it is typically run several times with different initializations, keeping the run with the smallest clustering error (Hastie et al., 2009, Sect. 14.3.6). A variant of $k$-means referred to as k-means++ uses a specific construction of the initial cluster centroids.

Lloyd's algorithm is a fixed-point iteration. Collect the cluster centroids in one vector ${\mathbf w} :=
\big({{\mathbf w}}^{(1)}, \ldots,
{{\mathbf w}}^{(k)}\big)$. One iteration, the assignment step Eq. (1) followed by the averaging step Eq. (2), applies an operator $\mathcal{F}$ to the current cluster centroids, so the method is the fixed-point iteration ${\mathbf w}^{(t+1)} =
\mathcal{F}\big({\mathbf w}^{(t)}\big)$ for $t = 0,1,\ldots$. The fixed points of $\mathcal{F}$ are cluster centroids that each coincide with the mean of the data points assigned to them. Convergence follows from monotone descent rather than from a contractive operator property: each of the two steps minimizes the clustering error over one set of variables with the other held fixed, so applying $\mathcal{F}$ never increases the clustering error. Non-increase alone does not exclude cycling through partitions of equal clustering error. If a data point changes its cluster centroid in the assignment step Eq. (1) only when another one is strictly closer, every change of the partition strictly lowers the clustering error, so no partition is visited twice. A finite dataset admits only finitely many partitions, so the partition stops changing after finitely many iterations, and the averaging step then leaves the cluster centroids unchanged: the iterates have reached a fixed point (Selim and Ismail, 1984). On the patches of Fig. 1, Lloyd's algorithm reached a fixed point after $12$ iterations, and the clustering error never increased along the way.

Lloyd's algorithm for $k$-means is the limit of the expectation–maximization algorithm for a Gaussian mixture model under additional constraints. Fix every cluster probability of the Gaussian mixture model at $1/k$ and every covariance matrix at $\sigma^{2} \mathbf{I}$, a multiple of the identity matrix with one common $\sigma^{2}$. The posterior distribution of the cluster index then depends on ${\mathbf x}$ only through its Euclidean distances to the means, and as $\sigma^{2}$ shrinks towards $0$ it concentrates on the closest one. In the limit ($\sigma^{2} \to 0$), the E-step of expectation–maximization becomes the assignment step Eq. (1) and the M-step becomes the averaging step Eq. (2) of Lloyd's algorithm (Bishop, 2006, Sect. 9.3.2). On the patches of Fig. 1 with $\sigma^{2} = 10^{-4}$, the expectation–maximization algorithm assigns $99.4\%$ of the patches like $k$-means. Leaving the covariance matrices and the cluster probabilities free is what lets a Gaussian mixture model provide a degree of belonging where $k$-means assigns each data point to one cluster.

The loss function is what separates $k$-means from its generalizations, and it decides what the averaging step of Lloyd's algorithm computes. Replacing the squared Euclidean distance by the Euclidean distance itself, $L\left({\mathbf x},h \right) = \left\lVert {\mathbf x} - h({\mathbf x}) \right\rVert_{2}$, gives the $k$-medians optimization problem. The assignment step Eq. (1) is unchanged, since the nearest cluster centroid is the same under both losses, but the averaging step Eq. (2) becomes the geometric median of the data points assigned to a cluster, which has no closed form and is computed by Weiszfeld's fixed-point iteration (Weiszfeld, 1937). The gain is robustness to outliers: adding $10$ far-away patches to the vegetation cluster of Fig. 1 moves their mean by $0.044$ but their geometric median by $0.004$, while $k$-medians assigns $98.5\%$ of the patches like $k$-means.

The averaging step is not tied to the squared Euclidean distance alone: for any Bregman divergence $L\left({\mathbf x},h \right) = D_{\phi}\big( {\mathbf x},h({\mathbf x}) \big)$ as loss, the mean of the assigned data points still minimizes their summed loss, so only the assignment step changes (Banerjee et al., 2005, Prop. 1). The Euclidean distance is not a Bregman divergence, which is why it changes the averaging step. Restricting each cluster centroid to be one of the data points assigned to it gives $k$-medoids, which uses only the pairwise distances between data points and therefore applies to data points without numeric feature vectors (Hastie et al., 2009, Sect. 14.3.10; Kaufman and Rousseeuw, 1990).

Returning to $k$-means itself, Fig. 3 shows the effect of the number $k$ of clusters on the patches of Fig. 1. Each patch is painted with the average color of its cluster, which is image segmentation. With $k = 2$, the sky and the mountain are separated from the vegetation. With $k = 3$, the mountain and haze form their own cluster. With $k = 4$, the vegetation splits into the sunlit meadow and the dark foliage. The clustering error decreases with $k$, from $0.034$ for $k = 2$ to $0.018$ for $k = 3$ and $0.011$ for $k = 4$, and it reaches $0$ when every data point is its own cluster centroid.

![Figure 3 of the entry kmeans](https://dictionaryofml.org/terms/fig/kmeans-fig3.svg)

*Figure 3: The photograph at patch resolution, each patch painted with the average color of its cluster under $k$-means with (a) $k = 2$, (b) $k = 3$, and (c) $k = 4$. In a grayscale print the clusters are told apart by brightness: the sky is brightest, mountain and haze are in between, and the vegetation is darkest; in (c) the sunlit meadow and the mountain are close in brightness. Data generated by pythondemos/kmeans.py*

**Synonyms:** k-means clustering.

**See also:** hard clustering, [clustering](https://dictionaryofml.org/terms/clustering.html), cluster, cluster centroid, clustering error, Lloyd's algorithm, fixed-point iteration, $k$-means++, Euclidean distance, [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), [hypothesis space](https://dictionaryofml.org/terms/hypospace.html), loss function, [Gaussian mixture model](https://dictionaryofml.org/terms/gmm.html), [expectation–maximization](https://dictionaryofml.org/terms/em.html), soft clustering, degree of belonging, geometric median, Bregman divergence.

## References

- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- MacQueen (1967). Some methods for classification and analysis of multivariate observations. *Proc. Fifth Berkeley Symp. Math. Statist. Probab.*.
- Mahajan et al. (2009). The Planar k-Means Problem is NP-Hard. *WALCOM: Algorithms and Computation*.
- Lloyd (1982). Least squares quantization in PCM. *IEEE Trans. Inf. Theory*. <https://doi.org/10.1109/TIT.1982.1056489>
- Selim and Ismail (1984). K-Means-Type Algorithms: A Generalized Convergence Theorem and Characterization of Local Optimality. *IEEE Trans. Pattern Anal. Mach. Intell.*. <https://doi.org/10.1109/TPAMI.1984.4767478>
- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Weiszfeld (1937). Sur le point pour lequel la somme des distances de $n$ points donnés est minimum. *Tôhoku Math. J.*.
- Banerjee et al. (2005). Clustering with Bregman Divergences. *J. Mach. Learn. Res.*. <https://jmlr.org/papers/v6/banerjee05b.html>
- Kaufman and Rousseeuw (1990). Finding Groups in Data: An Introduction to Cluster Analysis. *Wiley*.

## Cite this entry

```bibtex
@misc{dictml_kmeans,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {$k$-means},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/kmeans.html}
}
```


---

# label

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/label.html) · [typeset PDF](https://dictionaryofml.org/terms/label.pdf) · [LaTeX](https://dictionaryofml.org/terms/label.tex) · [Python demo](https://dictionaryofml.org/terms/label.py) · [notebook](https://dictionaryofml.org/terms/label.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A label is an attribute of a data point that represents a quantity of interest and that typically requires human expertise to determine. Predicting the label of a data point from its features is the goal of supervised learning. Determining the label of a data point is error-prone and often involves human annotation. The choice of label determines the learning task: whether the task is classification or regression depends on the label space together with the loss function. In self-supervised learning, the label is constructed from the features themselves, so that no manual annotation is required.

## Definition

For a data point that represents a patient, a feature could be the body weight, while the label could be the presence of a disease. A label of a data point ${\mathbf z}$ is one of its attributes that represents a higher-level fact, or quantity of interest, and that typically requires human expertise or domain knowledge to determine (Anon., 2003; Everitt and Skrondal, 2010; Gujarati and Porter, 2009). Unlike a feature, which can be computed or measured easily, a label is usually harder to obtain.

Whether a given attribute serves as a feature or a label is often a design choice that depends on the resources available in a given machine learning application. If an oncologist is readily available, a cancer prognosis can serve as a feature. On the other hand, if no oncologist is available, the cancer prognosis is a label that needs to be predicted from patient features.

The notion of a label is made precise by introducing a labeling function $\bar{h}: \mathcal{Z} \rightarrow \mathcal{Y}$. The domain of this function is the data point space $\mathcal{Z}$, the set of all possible data points that can occur in a given machine learning application. Its values lie in the label space $\mathcal{Y}$, the set of all possible values the label of a data point can take on. The function assigns each data point ${\mathbf z}$ its label $y = \bar{h}({\mathbf z})$ (Shalev-Shwartz and Ben-David, 2014, Sect. 2.1).

The labeling function $\bar{h}$ acts on the data point ${\mathbf z}$ itself, that is, on the whole patient. A trained model (or learned hypothesis) $\widehat{h}$ instead acts on the feature vector ${\mathbf x}$, a fixed list of recorded features such as the patient's body weight and age. The feature vector usually records only some attributes of a data point. The data point space $\mathcal{Z}$ contains patients, whereas the feature space $\mathcal{X}$ contains only their recorded feature vectors. Two patients with the same feature vector may therefore carry different labels, since they can differ in information that was never recorded as a feature. No hypothesis $\widehat{h}$ that depends on ${\mathbf x}$ alone can then deliver perfect predictions.

In practice, the true label $\bar{h}({\mathbf z})$ of a data point is rarely known exactly. Even with full access to the data point, its label must be determined by human judgment or measurement, which is imperfect. The available label is then a noisy version of $\bar{h}({\mathbf z})$, as in the additive error model of Hastie et al. (2009, Sect. 2.6).

For instance, an object category assigned by a human annotator or a diagnosis obtained from a clinician may be wrong, even though the annotator or clinician examines the data point directly. The discrepancy between such an observed label and the true label is called label noise and can degrade training and raise the risk of the learned hypothesis (Goodfellow et al., 2016, Sect. 7.5.1). This source of error differs from the incompleteness of the recorded features discussed above: here the data point is accessible, but its label is determined imperfectly.

One way to reduce the effect of label noise is label smoothing, which slightly perturbs the labels of data points used for model training (Goodfellow et al., 2016, Sect. 7.5.1). It is a form of data augmentation that acts as regularization.

Label noise also matters for regulation. The EU AI Act requires the training set of a high-risk artificial intelligence system to be, to the best extent possible, free of errors (European Parliament and Council of the European Union, 2024, Art. 10(3)) and the system to achieve an appropriate level of accuracy (European Parliament and Council of the European Union, 2024, Art. 15(1)); erroneous labels are one such error. When labels are personal data, the general data protection regulation requires them to be accurate (European Parliament and Council of the European Union, 2016, Art. 5(1)(d)).

Beyond how reliably a label is determined, which attribute is taken as the label, i.e., the choice of the labeling function $\bar{h}$, determines the learning task. For the image of a cow herd in Fig. 1, the label could be a binary indicator of whether the image contains cows (binary classification). Other choices are the number of cows or the average green level (regression), and a masked center pixel predicted from the surrounding pixels (self-supervised learning).

![Figure 1 of the entry label](https://dictionaryofml.org/terms/fig/label-fig1.svg)

*Figure 1: The same data point yields different learning tasks depending on the choice of label*

Whether a learning task is a classification problem or a regression problem depends not only on the label space $\mathcal{Y}$ but also on the loss function. For example, the label space $\mathcal{Y} = \{0,1,2,3,\ldots\}$ can be used for a classification method that uses the 0/1 loss as the loss function. The same label space can also be used for a regression method that uses the squared error loss as the loss function.

In self-supervised learning, the label is constructed from the features themselves, so no manual annotation is required. A prominent example is large language model training: consider the sentence ``All human beings are born free and equal,'' tokenized as the sequence (``All'', ``human'', ``beings'', ``are'', ``born'', ``free'', ``and'', ``equal''). The label for the subsequence (``All'', ``human'', ``beings'') is the next token ``are,'' the label for (``All'', ``human'', ``beings'', ``are'') is ``born,'' and so on. Taking every prefix of the sequence in turn, a single sentence produces many labeled data points from plain text alone.

**Synonyms:** target, response, ground truth, quantity of interest.

**See also:** [data point](https://dictionaryofml.org/terms/datapoint.html), [feature](https://dictionaryofml.org/terms/feature.html), label space, learning task, [classification](https://dictionaryofml.org/terms/classification.html), regression, [self-supervised learning](https://dictionaryofml.org/terms/selfsupervisedlearning.html), [data augmentation](https://dictionaryofml.org/terms/dataaug.html).

## References

- Anon. (2003). The Oxford Dictionary of Statistical Terms. *Oxford Univ. Press*. <https://doi.org/10.1093/oso/9780198509943.001.0001>
- Everitt and Skrondal (2010). The Cambridge Dictionary of Statistics. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9780511779633>
- Gujarati and Porter (2009). Basic Econometrics. *McGraw-Hill/Irwin*.
- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Goodfellow et al. (2016). Deep Learning. *MIT Press*.
- European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act) (Text with EEA relevance). <https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng>
- European Parliament and Council of the European Union (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (Text with EEA relevance). <https://eur-lex.europa.eu/eli/reg/2016/679/oj>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>

## Cite this entry

```bibtex
@incollection{dictml_label,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {label},
  booktitle = {Dictionary of Applied Machine Learning},
  publisher = {Springer Nature Singapore},
  year = {2026},
  doi = {10.1007/978-981-92-2012-0_12-1},
  isbn = {978-981-92-2012-0},
  url = {https://doi.org/10.1007/978-981-92-2012-0_12-1}
}
```


---

# learning rate

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-10-29. [HTML](https://dictionaryofml.org/terms/learnrate.html) · [typeset PDF](https://dictionaryofml.org/terms/learnrate.pdf) · [LaTeX](https://dictionaryofml.org/terms/learnrate.tex) · [Python demo](https://dictionaryofml.org/terms/learnrate.py) · [notebook](https://dictionaryofml.org/terms/learnrate.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A learning rate is a hyperparameter of an iterative machine learning method that controls the extent of a single update of the current hypothesis. A prime example for a learning rate is the step size of gradient descent. A learning rate that is too small makes little progress within the available number of iterations, and one that is too large can result in divergent iterations. For a fixed number of iterations, the learning rate bounds the effective hypothesis space of a machine learning method, which makes it a means of regularization. The learning rate can be chosen by the validation error of the learned hypothesis.

## Definition

Learning a linear model by gradient descent on the average squared error loss over a training set, as in Fig. 1, is an iterative machine learning method for learning a useful hypothesis $h \in
\mathcal{H}$. Such a method repeats similar update steps that modify the current hypothesis to obtain an improved one. A learning rate is a parameter of such a method that controls the extent to which the current hypothesis can be modified during a single iteration.

The main example of a learning rate $\eta$ is the step size of gradient descent (Boyd and Vandenberghe, 2004, Sect. 9.3),

\begin{equation}
\tag{1}
{\mathbf w}^{(t\!+\!1)} = {\mathbf w}^{(t)} - \eta \nabla f({\mathbf w}^{(t)}) \text{.}
\end{equation}

Here, the objective function $f({\mathbf w})$ is the empirical risk incurred by a hypothesis $h^{({\mathbf w})}$ on a training set, and each gradient step moves the model parameters against its gradient $\nabla f({\mathbf w}^{(t)})$ by a distance proportional to $\eta$. Fig. 1 illustrates one gradient step Eq. (1) of gradient descent for a linear model, started from the same hypothesis with three choices of the learning rate. The learning rates are given as multiples of $1/L$, with $L$ the largest eigenvalue of the Hessian of $f$. For the squared error loss, a gradient step lowers $f$ for every learning rate below $2/L$ (Nesterov, 2004, Thm. 2.1.14); the largest learning rate of Fig. 1 lies above that threshold, and its step raises $f$.

![Figure 1 of the entry learnrate](https://dictionaryofml.org/terms/fig/learnrate-fig1.svg)

*Figure 1: One gradient step of gradient descent on the average squared error loss of a linear model on a training set. The solid line is the hypothesis map before the step. The dashed line is the hypothesis map after one step with the learning rate $\eta = 0.1/L$, the dotted line after one step with $\eta = 1.5/L$, and the gray dash-dotted line after one step with $\eta = 2.5/L$, where $L$ is the largest eigenvalue of the Hessian of the objective function. Data generated by pythondemos/learnrate.py*

Each iteration of a machine learning method should make large progress towards a useful hypothesis, so the learning rate should not be too small. The step with $\eta = 2.5/L$ in Fig. 1 shows the other failure: a too large step size of gradient descent makes the gradient steps overshoot a minimum of the objective function. Fig. 2 illustrates both cases.

![Figure 2 of the entry learnrate](https://dictionaryofml.org/terms/fig/learnrate-fig2.svg)

*Figure 2: Effect of using an inadequate step size $\eta$ in the gradient step Eq. (1). (a) If $\eta$ is too large, the gradient steps can overshoot such that the iterates ${\mathbf w}^{(t)}$ move away from a minimum of $f$, i.e., $f({\mathbf w}^{(t\!+\!1)}) > f({\mathbf w}^{(t)})$. (b) If $\eta$ is too small, the gradient steps make too little progress towards a minimum within the available number of iterations*

Consider an iterative machine learning method that learns a hypothesis from some underlying hypothesis space $\mathcal{H}$ by running a fixed number $T$ of iterations. Varying the learning rate implements regularization by pruning the effective $\mathcal{H}$. For example, each gradient step above Eq. (1) moves the model parameters by $\eta \Vert {\nabla f({\mathbf w}^{(t)})} \Vert$. By the triangle inequality,

\begin{equation}
\tag{2}
\Vert {{\mathbf w}^{(T)} - {\mathbf w}^{(0)}} \Vert \leq \eta
\sum_{t=0}^{T-1} \Vert {\nabla f({\mathbf w}^{(t)})} \Vert \text{.}
\end{equation}

Together with $T$, the learning rate thus determines the region of the parameter space that the method can reach, and thereby the effective hypothesis space: the hypotheses $h^{({\mathbf w})}$ whose model parameters ${\mathbf w}$ lie in that region.

Fig. 3 draws the balls around ${\mathbf w}^{(0)}$ with the radius Eq. (2) for $T = 10$ gradient steps of gradient descent solving the empirical risk minimization problem of linear regression, and three learning rates. Reducing the learning rate shrinks the effective hypothesis space that can be reached by the gradient steps. Choosing the learning rate too small or too large can then result in underfitting or overfitting.

![Figure 3 of the entry learnrate](https://dictionaryofml.org/terms/fig/learnrate-fig3.svg)

*Figure 3: The model parameters that $T = 10$ gradient steps for linear regression can reach from the initial ${\mathbf w}^{(0)}$ (filled square); each ball is the effective hypothesis space $\mathcal{H}^{(\eta)}$ of its learning rate. Each curve is the ball around ${\mathbf w}^{(0)}$ with the radius Eq. (2), computed along the iterations with the learning rate $\eta = 0.1/L$ (solid curve), $0.5/L$ (dashed curve) and $1.5/L$ (dotted curve). The empirical risk minimization solution $\widehat{{\mathbf w}}$ (star) lies outside the two smaller balls and inside the largest one. Data generated by pythondemos/learnrate.py*

As a hyperparameter, the learning rate can be chosen by validation: the hypotheses learned with different learning rates are compared by their validation error on a validation set.

**Synonyms:** step size.

**See also:** machine learning, [hypothesis](https://dictionaryofml.org/terms/hypothesis.html), parameter, [gradient descent](https://dictionaryofml.org/terms/gd.html), [stochastic gradient descent](https://dictionaryofml.org/terms/stochGD.html), projected gradient descent, step size, [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), linear model, [hyperparameter](https://dictionaryofml.org/terms/hyperparameter.html), parameter space, [hypothesis space](https://dictionaryofml.org/terms/hypospace.html).

## References

- Boyd and Vandenberghe (2004). Convex Optimization. *Cambridge Univ. Press*. <https://doi.org/10.1017/CBO9780511804441>
- Nesterov (2004). Introductory Lectures on Convex Optimization: A Basic Course. *Kluwer Academic*. <https://doi.org/10.1007/978-1-4419-8853-9>

## Cite this entry

```bibtex
@misc{dictml_learnrate,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {learning rate},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/learnrate.html}
}
```


---

# linear classifier

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-07. [HTML](https://dictionaryofml.org/terms/linclass.html) · [typeset PDF](https://dictionaryofml.org/terms/linclass.pdf) · [LaTeX](https://dictionaryofml.org/terms/linclass.tex) · [Python demo](https://dictionaryofml.org/terms/linclass.py) · [notebook](https://dictionaryofml.org/terms/linclass.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A linear classifier is a classifier whose decision regions are separated by hyperplanes. Each linear classifier is obtained by comparing the value of a linear hypothesis map against a threshold. For binary classification, the sign of the value can be used to determine the predicted label, and the absolute value measures the distance of the data point from the decision boundary. The model parameters of a linear classifier are the weight vector of the underlying hypothesis map and the threshold. They are learned by empirical risk minimization with different loss functions: logistic regression uses the logistic loss and the support vector machine the hinge loss. For multi-class classification, a linear classifier is characterized by several linear model hypothesis maps, one per label value, and the resulting decision regions are intersections of halfspaces.

## Definition

An aerial photograph of a wine-growing region is divided into small areas, and each area is to be classified as vineyard or not. Each area is a data point characterized by two features obtained from the photograph (Fig. 1). The label of each area indicates whether the area belongs to a vineyard ($y = 1$) or not ($y = -1$). The labeled areas form the training set. A linear classifier separates the two classes by a straight line, its decision boundary, which is fixed by a normal vector, and it classifies a new area by its signed distance from that line.

![Figure 1 of the entry linclass](https://dictionaryofml.org/terms/fig/linclass-fig1.svg)

*Figure 1: Scatterplot of data points, each characterized by two features and a binary label. The solid line is the decision boundary $h({\mathbf x}) = {\mathbf w}^{\top} {\mathbf x} + b = 0$ of a linear classifier learned from these data points, and the single-headed arrow is its normal vector ${\mathbf w}$. The double-headed arrow spans $|h({\mathbf x})| / \left\lVert {\mathbf w} \right\rVert_{2}$, the distance from the decision boundary of a new data point ${\mathbf x}$ (open square) that is not in the training set. Data generated by pythondemos/linclass.py*

A linear classifier is a classifier whose decision regions are separated by hyperplanes in $\mathbb{R}^{d}$ (Bishop, 2006, Sect. 4.1.1; Shalev-Shwartz and Ben-David, 2014, Sect. 9.1). For binary classification, with label space $\mathcal{Y} = \{-1, +1\}$, each linear classifier can be written as the thresholding rule $\widehat{y} =
\operatorname{sign}(h({\mathbf x}))$ with the hypothesis

$$
h({\mathbf x}) = {\mathbf w}^{\top} {\mathbf x}
+ b \text{.}
$$

The hypothesis map is parameterized by the weight vector ${\mathbf w} \in \mathbb{R}^{d} \setminus \{\mathbf{0}\}$ and the threshold $b \in \mathbb{R}$. A data point with feature vector ${\mathbf x}$ is classified as $\widehat{y} =
1$ if $h({\mathbf x}) \geq 0$ and as $\widehat{y} =
-1$ otherwise. The resulting decision boundary is the hyperplane

$$
\{ {\mathbf x} \in \mathbb{R}^{d} : {\mathbf w}^{\top}
{\mathbf x} + b = 0 \} \text{,}
$$

with normal vector ${\mathbf w}$. This boundary separates two halfspaces, which are the decision regions $\mathcal{R}_{1} = \{ {\mathbf x} : h({\mathbf x}) \geq 0 \}$ and $\mathcal{R}_{-1} = \{ {\mathbf x} : h({\mathbf x}) < 0 \}$.

Consider a feature vector ${\mathbf x}$ and any vector ${\mathbf x}'$ on the decision boundary, $h({\mathbf x}') = 0$. Subtracting ${\mathbf w}^{\top} {\mathbf x}' + b = 0$ from $h({\mathbf x})$ gives

$$
h({\mathbf x}) = {\mathbf w}^{\top}
\big( {\mathbf x} - {\mathbf x}' \big) \text{.}
$$

The Cauchy-Schwarz inequality bounds the right-hand side in magnitude by $\left\lVert {\mathbf w} \right\rVert_{2} \left\lVert {\mathbf x} -
{\mathbf x}' \right\rVert_{2}$ (Axler, 2024, Thm. 6.14). Every vector of the decision boundary is therefore at distance at least $|h({\mathbf x})| / \left\lVert {\mathbf w} \right\rVert_{2}$ from ${\mathbf x}$. That distance is attained by the orthogonal projection

$$
{\mathbf x}' = {\mathbf x} - \frac{h({\mathbf x})}
{\left\lVert {\mathbf w} \right\rVert_{2}^{2}} \, {\mathbf w} \text{.}
$$

The distance of ${\mathbf x}$ from the decision boundary is thus $|h({\mathbf x})| / \left\lVert {\mathbf w} \right\rVert_{2}$, and the hypothesis value $h({\mathbf x})$ is that distance multiplied by $\left\lVert {\mathbf w} \right\rVert_{2}$ (Bishop, 2006, Sect. 4.1.1; Hastie et al., 2009, Sect. 4.5, Eq. (4.40)). Fig. 1 illustrates the distance to the decision boundary for one data point.

The value $|h({\mathbf x})|$ is a measure of the noise tolerance at the feature vector ${\mathbf x}$. Perturbing ${\mathbf x}$ by a vector $\boldsymbol{\delta}$ changes the hypothesis value by ${\mathbf w}^{\top} \boldsymbol{\delta}$, which the Cauchy-Schwarz inequality bounds in magnitude by $\left\lVert {\mathbf w} \right\rVert_{2} \left\lVert \boldsymbol{\delta} \right\rVert_{2}$. The sign of $h({\mathbf x})$, and with it the prediction, is therefore unchanged by every perturbation with $\left\lVert \boldsymbol{\delta} \right\rVert_{2} < |h({\mathbf x})|
/ \left\lVert {\mathbf w} \right\rVert_{2}$.

Multiplying $h({\mathbf x})$ by the label $y$ gives the margin. A positive margin means that the data point is correctly classified, a negative margin that it is misclassified, and a zero margin that it lies on the decision boundary, where the rule above assigns $\widehat{y} = 1$.

The model parameters $({\mathbf w}, b)$ are learned from a training set $\mathcal{D}^{(\rm train)} = \{ ({\mathbf x}^{(r)},
y^{(r)}) \}_{r = 1}^{m}$ by empirical risk minimization. Using the 0/1 loss results in an objective function that is piecewise constant, and minimizing it is computationally hard in general (Shalev-Shwartz and Ben-David, 2014, Sect. 9.1). Practical methods replace the 0/1 loss by a convex loss of the margin: logistic regression uses the logistic loss and the support vector machine uses the hinge loss. For a linearly separable training set and a sufficiently small regularization parameter $\alpha$, the support vector machine learns the linear classifier whose smallest margin is largest (Shalev-Shwartz and Ben-David, 2014, Sect. 15.1, Eq. (15.1)). Both learn the model parameters of a linear classifier by minimizing a convex objective function: gradient descent minimizes the smooth objective function of logistic regression, and subgradient descent the non-smooth objective function of the support vector machine (Murphy, 2012, Sect. 8.3.2; Shalev-Shwartz and Ben-David, 2014, Sect. 14.2.3, Example 14.2).

For multi-class problems, with label space $\mathcal{Y} = \{1,
\ldots, K\}$ and $K > 2$, a linear classifier uses one linear model hypothesis $g_{c}({\mathbf x}) = \big( {\mathbf w}^{(c)}
\big)^{\top} {\mathbf x} + b^{(c)}$ per label value $c \in \mathcal{Y}$ and predicts the value with the largest score, $\widehat{y} = \mathop{\mathrm{arg\,max}}_{c \in \mathcal{Y}}
g_{c}({\mathbf x})$. The decision region $\mathcal{R}_{c}$ is then an intersection of $K - 1$ halfspaces and is therefore a convex set. The border between two decision regions $\mathcal{R}_{c}$ and $\mathcal{R}_{c'}$ lies in the hyperplane on which the two scores agree (Bishop, 2006, Sect. 4.1.2, Eq. (4.10)). The resulting partition of the feature space is shown in Fig. 2 for a linear classifier with $K = 3$ classes.

![Figure 2 of the entry linclass](https://dictionaryofml.org/terms/fig/linclass-fig2.svg)

*Figure 2: A linear classifier for $K = 3$ label values partitions the feature space $\mathbb{R}^{2}$ into three decision regions $\mathcal{R}_{1}, \mathcal{R}_{2}, \mathcal{R}_{3}$. Each border lies in the hyperplane on which the two competing scores agree, and each decision region is an intersection of halfspaces and therefore convex*

A linear classifier can be generalized to allow for non-linear decision boundaries by using a feature transformation. Applying a feature transformation $\phi:
\mathbb{R}^{d} \to \mathbb{R}^{d'}$ first and a linear classifier to $\phi({\mathbf x})$ afterwards gives a decision boundary that is a hyperplane in $\mathbb{R}^{d'}$ but non-linear in the original feature space. A kernel method constructs such a feature transformation from a kernel, and a deep net uses its first layers as a trainable feature transformation (Bishop and Bishop, 2024, Sect. 6.3.3; Schölkopf and Smola, 2002, Sect. 2.2).

**See also:** classifier, [binary classification](https://dictionaryofml.org/terms/binclass.html), [decision boundary](https://dictionaryofml.org/terms/decisionboundary.html), [decision region](https://dictionaryofml.org/terms/decisionregion.html), hyperplane, halfspace, linear model, margin, [logistic regression](https://dictionaryofml.org/terms/logreg.html), [support vector machine](https://dictionaryofml.org/terms/svm.html), [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), $\bf 0/1$ loss, feature transformation.

## References

- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Axler (2024). Linear Algebra Done Right. *Springer*. <https://doi.org/10.1007/978-3-031-41026-0>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Murphy (2012). Machine Learning: A Probabilistic Perspective. *MIT Press*.
- Bishop and Bishop (2024). Deep Learning: Foundations and Concepts. *Springer Nature*. <https://doi.org/10.1007/978-3-031-45468-4>
- Schölkopf and Smola (2002). Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. *MIT Press*. <https://doi.org/10.7551/mitpress/4175.001.0001>

## Cite this entry

```bibtex
@misc{dictml_linclass,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {linear classifier},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/linclass.html}
}
```


---

# linear regression

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/linreg.html) · [typeset PDF](https://dictionaryofml.org/terms/linreg.pdf) · [LaTeX](https://dictionaryofml.org/terms/linreg.tex) · [Python demo](https://dictionaryofml.org/terms/linreg.py) · [notebook](https://dictionaryofml.org/terms/linreg.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Linear regression is a regression method that learns a linear hypothesis map for predicting the numeric label of a data point from its features. Its least squares variant chooses a map that minimizes the average squared error loss on a training set. Linear regression is an instance of empirical risk minimization that is obtained by using the linear model and the squared error loss. The optimal model parameters are determined by the normal equations. This system can be solved in closed form or iteratively, for example, by gradient descent. Adding a penalty term to the average squared error loss yields regularized variants such as ridge regression and least absolute shrinkage and selection operator.

## Definition

Linear regression can be used to predict tomorrow's temperature from weather measurements recorded over the last five days, using them as features and tomorrow's temperature as the label. Linear regression methods use a linear model: they learn a linear hypothesis map that delivers a prediction of the numeric label of a data point. The prediction is based solely on the features of the data point, which are fed as input to the learned hypothesis map.

Formally, linear regression learns a linear hypothesis map $h^{({\mathbf w})}({\mathbf x}) = {\mathbf w}^{\top}{\mathbf x}$ to predict the numeric label $y \in \mathbb{R}$ of a data point from its feature vector ${\mathbf x} = \big(x_{1},\,\ldots,\,x_{d}\big)^{\top} \in \mathbb{R}^{d}$. Here, $d$ denotes the number of features of a data point. The model parameters ${\mathbf w}$ are learned from a training set $\mathcal{D}^{(\rm train)} = \big\{ \left( {\mathbf x}^{(r)},y^{(r)} \right) \big\}_{r=1}^{m}$ of data points. Appending a constant feature to ${\mathbf x}$ lets one entry of ${\mathbf w}$ act as an intercept, so ${\mathbf w}^{\top}{\mathbf x}$ represents an affine function of the original measurements. Below, ${\mathbf x}$ denotes whichever feature vector is fed to the linear hypothesis map, original or augmented, unless stated otherwise.

The least squares variant of linear regression measures the quality of a linear hypothesis map by its average loss, the squared error loss, on the training set. As an instance of empirical risk minimization, it learns the model parameters ${\mathbf w}$ by solving the optimization problem

$$
\min_{{\mathbf w} \in \mathbb{R}^{d}} \frac{1}{m}
\sum_{r=1}^{m}
\big( y^{(r)} - {\mathbf w}^{\top} {\mathbf x}^{(r)} \big)^{2} \text{.}
$$

Fig. 1 illustrates this linear regression problem for a training set with a constant scalar feature $x^{(r)}=1$ for $r=1,\ldots,m$.

![Figure 1 of the entry linreg](https://dictionaryofml.org/terms/fig/linreg-fig1.svg)

*Figure 1: For a linear model with $d=1$ and the constant feature $x=1$ for every data point (blue filled circles), linear regression reduces to computing the average $\widehat{w} = (1/m) \sum_{r=1}^{m} y^{(r)}$ of the labels: this average minimizes the average squared error loss. The horizontal dashed line sits at the height $\widehat{w}$, and the vertical gray segments are the errors $y^{(r)} - \widehat{w}$ entering that loss*

The optimization problem can be written more compactly using the feature matrix ${\mathbf X}$ and the label vector ${\mathbf y}$,

$$
{\mathbf X} = \big({\mathbf x}^{(1)},\,\ldots,\,{\mathbf x}^{(m)}\big)^{\top}
\in \mathbb{R}^{m \times d} \text{,} \qquad
{\mathbf y} = \big( y^{(1)},\,\ldots,\,y^{(m)} \big)^{\top}
\in \mathbb{R}^{m} \text{.}
$$

In terms of ${\mathbf X}$ and ${\mathbf y}$, the optimization problem reads

$$
\min_{{\mathbf w} \in \mathbb{R}^{d}} \underbrace{\frac{1}{m}
\left\lVert {\mathbf y} - {\mathbf X} {\mathbf w} \right\rVert_{2}^{2}}_{f({\mathbf w})} \text{.}
$$

The average squared error loss $f({\mathbf w})$ is a convex and differentiable function of ${\mathbf w}$. By the zero-gradient condition (Boyd and Vandenberghe, 2004, Sect. 4.2.3), a vector $\widehat{{\mathbf w}}$ solves this optimization problem if and only if it satisfies the system of linear equations (Strang, 2016)

\begin{equation}
\tag{1}
{\mathbf X}^{\top}{\mathbf X} \widehat{{\mathbf w}} = {\mathbf X}^{\top} {\mathbf y} \text{.}
\end{equation}

The normal equations Eq. (1) always have a solution because ${\mathbf X}^{\top}{\mathbf y}$ lies in the column space of ${\mathbf X}^{\top}$, which coincides with the column space of ${\mathbf X}^{\top}{\mathbf X}$ (Meyer, 2000, Sect. 4.5).

However, the solution of the normal equations Eq. (1) is unique only if the feature matrix ${\mathbf X}$ has full column rank. This requires at least as many data points as features, $m \geq d$. When this holds, and the columns of ${\mathbf X}$ are linearly independent, the matrix ${\mathbf X}^{\top}{\mathbf X}$ is invertible and the normal equations Eq. (1) have the unique closed-form solution $\widehat{{\mathbf w}} = \big({\mathbf X}^{\top}{\mathbf X}\big)^{-1} {\mathbf X}^{\top} {\mathbf y}$ (Golub and Loan, 2013, Sect. 5.3.1).

If instead $m < d$ (more features than data points), ${\mathbf X}$ cannot have full column rank, ${\mathbf X}^{\top}{\mathbf X}$ is singular, and the optimization problem has infinitely many solutions (Golub and Loan, 2013, Sect. 5.5). All of them incur the same minimal average squared error loss on the training set, but their predictions for data points outside the training set can differ arbitrarily. Choosing among them is therefore a matter of generalization (see overfitting). A unique solution can be selected, for example, by picking the minimum-norm solution ${{\mathbf X}}^{+} {\mathbf y}$ given by the pseudoinverse ${{\mathbf X}}^{+}$ (Golub and Loan, 2013, Sect. 5.5.2).

A common remedy is regularization: adding a penalty term to the average squared error loss. The penalty term can be interpreted as an estimate of how much higher the loss is on data points outside the training set than on it (Hastie et al., 2009, Sect. 7.4). This yields regularized variants of linear regression. Ridge regression uses the penalty term $\alpha \left\lVert {\mathbf w} \right\rVert_{2}^{2}$ and the least absolute shrinkage and selection operator uses $\alpha \left\lVert {\mathbf w} \right\rVert_{1}$. The hyperparameter $\alpha > 0$ sets the weight of the penalty term.

Linear regression also has a statistical interpretation. Consider a probabilistic model with a joint probability distribution $\mathbb{P}^{({\mathbf x}, y)}$ over the features and the label. Under the squared error loss, the Bayes estimator (i.e., the hypothesis with minimum risk) is the conditional expectation $\mathbb{E}\{y \mid {\mathbf x}\}$ (Lehmann and Casella, 1998, Ch. 4; Papoulis and Pillai, 2002). In this statistical interpretation, ${\mathbf x}$ denotes the original feature vector. Let the features and the label be jointly Gaussian RVs with zero mean and invertible covariance matrix $\mathbf{C}^{({\mathbf x})} = \mathbb{E}\{{\mathbf x} {\mathbf x}^{\top}\}$. This Bayes estimator is then (Bishop, 2006, Sect. 2.3.1, Eq. (2.81))

$$
h^{*}({\mathbf x}) = \big({\mathbf w}^{\star}\big)^{\top}{\mathbf x}
\text{, with } {\mathbf w}^{\star} = \big(\mathbf{C}^{({\mathbf x})}\big)^{-1}\, \mathbf{c}^{({\mathbf x},y)} \text{.}
$$

Here, $\mathbf{c}^{({\mathbf x},y)} = \mathbb{E}\{{\mathbf x} y\}$ is the covariance between the features ${\mathbf x}$ and the label $y$. This Bayes estimator is linear in ${\mathbf x}$. Without the zero-mean assumption, it is an affine function of ${\mathbf x}$, which a constant feature absorbs as before.

Least-squares linear regression is the empirical risk minimization counterpart of this predictor. It replaces $\mathbf{C}^{({\mathbf x})}$ and $\mathbf{c}^{({\mathbf x},y)}$ with the sample-based estimates $(1/m){\mathbf X}^{\top}{\mathbf X}$ and $(1/m){\mathbf X}^{\top}{\mathbf y}$. For invertible ${\mathbf X}^{\top}{\mathbf X}$, this recovers the unique solution $\widehat{{\mathbf w}} = \big({\mathbf X}^{\top}{\mathbf X}\big)^{-1}{\mathbf X}^{\top}{\mathbf y}$ of the normal equations Eq. (1).

Instead of solving the normal equations Eq. (1) directly (via the inverse or pseudoinverse), machine learning methods often solve it by an iterative optimization method, as implemented in widely used machine learning software libraries such as scikit-learn (Pedregosa et al., 2011) and PyTorch (Paszke et al., 2019). Starting from initial model parameters ${\mathbf w}^{(0)}$, such a method repeatedly applies an update operator $\mathcal{F}$,

$$
{\mathbf w}^{(t+1)} = \mathcal{F}\big({\mathbf w}^{(t)}\big)
\text{, for } t = 0,1,\ldots \text{.}
$$

The operator $\mathcal{F}$ is designed such that the sequence of model parameters ${\mathbf w}^{(t)}$ converges to a solution of the normal equations Eq. (1).

One such iterative optimization method is gradient descent, which uses the gradient descent step operator

$$
\mathcal{F}^{(\eta)}({\mathbf w}) = {\mathbf w} - \eta \nabla f({\mathbf w}) \text{.}
$$

Here, $\eta > 0$ is a step size that must be chosen small enough to ensure convergence (Bertsekas, 2016, Prop. 1.2.2). The superscript in $\mathcal{F}^{(\eta)}$ makes the dependence of the operator on $\eta$ explicit. Inserting the average training error into the general form of the gradient descent step yields the explicit update

$$
\begin{aligned}
\mathcal{F}^{(\eta)}({\mathbf w})
&= {\mathbf w} + \frac{2\eta}{m} \sum_{r=1}^{m}
\big( y^{(r)} - {\mathbf w}^{\top}{\mathbf x}^{(r)} \big)
{\mathbf x}^{(r)} \\
&= \Big(\mathbf{I} - \frac{2\eta}{m}{\mathbf X}^{\top}{\mathbf X}\Big){\mathbf w}
+ \frac{2\eta}{m}{\mathbf X}^{\top}{\mathbf y} \text{.}
\end{aligned}
$$

The fixed points of $\mathcal{F}^{(\eta)}$ are exactly the solutions of the normal equations Eq. (1), since $\mathcal{F}^{(\eta)}(\widehat{{\mathbf w}}) = \widehat{{\mathbf w}}$ holds if and only if ${\mathbf X}^{\top}{\mathbf X}\widehat{{\mathbf w}} = {\mathbf X}^{\top}{\mathbf y}$. The update operator is affine,

$$
\mathcal{F}^{(\eta)}({\mathbf w}) = {\mathbf M} {\mathbf w} + {\mathbf b} \text{,}
$$

with linear part ${\mathbf M} = \mathbf{I} - (2\eta/m){\mathbf X}^{\top}{\mathbf X}$ and offset ${\mathbf b} = (2\eta/m)\, {\mathbf X}^{\top}{\mathbf y}$.

When ${\mathbf X}$ has full column rank, so that ${\mathbf X}^{\top}{\mathbf X}$ is positive definite, $\mathcal{F}^{(\eta)}$ is a contractive operator with respect to the Euclidean norm for every step size $0 < \eta < m/\lambda_{\max}$. Here, $\lambda_{\max}$ is the largest eigenvalue of ${\mathbf X}^{\top}{\mathbf X}$. The eigenvalues of the linear part ${\mathbf M}$ are $1 - (2\eta/m)\,\lambda_{j}$, with the positive eigenvalues $\lambda_{j}$ of ${\mathbf X}^{\top}{\mathbf X}$. They lie strictly between $-1$ and $1$ exactly for such step sizes. The more general case of any smooth, strongly convex objective function and step sizes $\eta \leq m/(\lambda_{\min} + \lambda_{\max})$ is covered by Nesterov (2004, Thm. 2.1.15), with the smallest eigenvalue $\lambda_{\min}$ of ${\mathbf X}^{\top}{\mathbf X}$. The fixed-point iteration converges to the unique solution $\widehat{{\mathbf w}}$. Moreover, the convergence speed of gradient descent is governed by the ratio of the largest to the smallest eigenvalue of ${\mathbf X}^{\top}{\mathbf X}$, that is, by its condition number $\kappa\!\left({\mathbf X}^{\top}{\mathbf X}\right) = \lambda_{\max}/\lambda_{\min}$ (Boyd and Vandenberghe, 2004, Sect. 9.3).

The gradient descent step above uses the entire training set to compute the gradient. When the data points instead arrive sequentially at time instants $t=1,2,\ldots$, or the training set is too large to fit in memory, the model parameters can be learned by online gradient descent. At each time step $t$, it applies a single gradient descent step. That step uses only the arriving data point $\left( {\mathbf x}^{(t)},y^{(t)} \right)$,

$$
{\mathbf w}^{(t+1)} = {\mathbf w}^{(t)} + 2\eta \big(
y^{(t)} - \big({\mathbf w}^{(t)}\big)^{\top}
{\mathbf x}^{(t)} \big) {\mathbf x}^{(t)} \text{.}
$$

This is the gradient descent step on the loss function $\big(y^{(t)} - {\mathbf w}^{\top}{\mathbf x}^{(t)}\big)^{2}$ of the arriving data point alone, rather than on the average squared error loss $f({\mathbf w})$. It is the least mean squares update (Bishop, 2006, Sect. 3.1.3). It avoids storing the entire training set, and its per-iteration complexity is independent of the training set size $m$.

The optimality condition Eq. (1) allows one to study the stability of linear regression. Ideally, the learned model parameters are insensitive to limited perturbations of the training set. A perturbed training set yields a feature matrix $\widetilde{{\mathbf X}} = {\mathbf X} + \Delta {\mathbf X}$ and a label vector $\widetilde{{\mathbf y}} = {\mathbf y} + \Delta {\mathbf y}$, with perturbation matrix $\Delta {\mathbf X}$ and vector $\Delta {\mathbf y}$. This gives the perturbed normal equations

\begin{equation}
\tag{2}
\widetilde{{\mathbf X}}^{\top} \widetilde{{\mathbf X}} \widetilde{{\mathbf w}}
= \widetilde{{\mathbf X}}^{\top} \widetilde{{\mathbf y}} \text{.}
\end{equation}

Matrix perturbation theory quantifies how much $\widetilde{{\mathbf w}}$ deviates from a solution $\widehat{{\mathbf w}}$ of the clean normal equations Eq. (1) (Golub and Loan, 2013, Sect. 5.3.6).

For a label-only perturbation, which replaces a single label of the training set with an outlier or another corrupted value, $\Delta {\mathbf X} = \mathbf{0}$, and the perturbed and clean normal equations share the coefficient matrix ${\mathbf X}^{\top}{\mathbf X}$. Assume ${\mathbf X}$ has full column rank, so ${\mathbf X}^{\top}{\mathbf X}$ is invertible and the solution is unique. Subtracting the clean normal equations Eq. (1) from the perturbed normal equations Eq. (2) yields

$$
\widetilde{{\mathbf w}} - \widehat{{\mathbf w}} = \big({\mathbf X}^{\top}{\mathbf X}\big)^{-1} {\mathbf X}^{\top} \Delta {\mathbf y} = {{\mathbf X}}^{+} \Delta {\mathbf y} \text{,}
$$

with the pseudoinverse ${{\mathbf X}}^{+}$. The effect of perturbing the training set is thereby quantified,

$$
\left\lVert \widetilde{{\mathbf w}} - \widehat{{\mathbf w}} \right\rVert_{2} \leq \left\lVert {{\mathbf X}}^{+} \right\rVert_{2}\, \left\lVert \Delta {\mathbf y} \right\rVert_{2} \text{.}
$$

When ${\mathbf X}$ has full column rank, the pseudoinverse has the norm $\left\lVert {{\mathbf X}}^{+} \right\rVert_{2} = 1/\sqrt{\lambda_{\min}({\mathbf X}^{\top}{\mathbf X})}
= \sqrt{\kappa\!\left({\mathbf X}^{\top}{\mathbf X}\right)}/\left\lVert {\mathbf X} \right\rVert_{2}$, where $\kappa\!\left({\mathbf X}^{\top}{\mathbf X}\right) = \lambda_{\max}/\lambda_{\min}$ is the condition number of ${\mathbf X}^{\top}{\mathbf X}$ (see pseudoinverse). In the setting of Fig. 1, ${\mathbf X} = \mathbf{1}$ (the all-ones vector), so ${{\mathbf X}}^{+} \Delta {\mathbf y} =
(1/m) \sum_{r=1}^{m} \Delta y^{(r)}$. Fig. 2 illustrates such a label-only perturbation of the training set from Fig. 1: a single label perturbation shifts $\widehat{w}$ by one third of it, as Fig. 2 shows.

![Figure 2 of the entry linreg](https://dictionaryofml.org/terms/fig/linreg-fig2.svg)

*Figure 2: The perturbation $\Delta y^{(3)} = 6$ (red arrow) replaces the third label of the training set from Fig. 1 with an outlier: filled circles denote the original data points, as in Fig. 1, and the open circle the perturbed value. The outlier raises the average of the labels: the learned model parameter shifts by $\widetilde{w} - \widehat{w} = 6/3 = 2$, from the clean solution $\widehat{w} = 3$ (dashed line) to the perturbed solution $\widetilde{w} = 5$ (dotted line)*

The effect of perturbations can also be studied on the level of a specific optimization method. For example, the update of online gradient descent becomes

$$
\begin{aligned}
{\mathbf w}^{(t+1)}
&= {\mathbf w}^{(t)} + 2\eta \big(
\widetilde{y}^{(t)} - \big({\mathbf w}^{(t)}\big)^{\top}
\widetilde{{\mathbf x}}^{(t)} \big) \widetilde{{\mathbf x}}^{(t)} \\
&= {\mathbf w}^{(t)} + 2\eta \big(
y^{(t)} - \big({\mathbf w}^{(t)}\big)^{\top}
{\mathbf x}^{(t)} \big) {\mathbf x}^{(t)} + {\boldsymbol \varepsilon}^{(t)} \text{.}
\end{aligned}
$$

Here, ${\boldsymbol \varepsilon}^{(t)}$ is a perturbation term that depends on the data point perturbation $\left( \Delta {\mathbf x}^{(t)},\Delta y^{(t)} \right)$ and the current model parameters ${\mathbf w}^{(t)}$. A bounded perturbation term moves each iterate by a bounded amount. The perturbed online gradient descent update has no fixed point in general, since a single perturbed data point leaves $\widetilde{{\mathbf w}}$ unchanged only if $\widetilde{{\mathbf w}}$ interpolates it. The gradient descent step operator $\mathcal{F}^{(\eta)}$ computed on the perturbed training set instead has as its fixed points exactly the solutions of the perturbed normal equations Eq. (2), by the same computation as for the clean training set. Among them is the perturbed solution $\widetilde{{\mathbf w}}$, whose displacement from $\widehat{{\mathbf w}}$ was computed above.

**See also:** regression, linear model, least squares, squared error loss, [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), [gradient descent](https://dictionaryofml.org/terms/gd.html), ridge regression, least absolute shrinkage and selection operator, pseudoinverse.

## References

- Boyd and Vandenberghe (2004). Convex Optimization. *Cambridge Univ. Press*. <https://doi.org/10.1017/CBO9780511804441>
- Strang (2016). Introduction to Linear Algebra. *Wellesley-Cambridge Press*.
- Meyer (2000). Matrix Analysis and Applied Linear Algebra. *SIAM*.
- Golub and Loan (2013). Matrix Computations. *The Johns Hopkins Univ. Press*.
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Lehmann and Casella (1998). Theory of Point Estimation. *Springer-Verlag*. <https://doi.org/10.1007/b98854>
- Papoulis and Pillai (2002). Probability, Random Variables, and Stochastic Processes. *McGraw-Hill Higher Education*.
- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Pedregosa et al. (2011). Scikit-learn: Machine Learning in Python. *Journal of Machine Learning Research*. <https://www.jmlr.org/papers/v12/pedregosa11a.html>
- Paszke et al. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. *Adv. Neural Inf. Process. Syst.*. <https://proceedings.neurips.cc/paper_files/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html>
- Bertsekas (2016). Nonlinear Programming. *Athena Scientific*.
- Nesterov (2004). Introductory Lectures on Convex Optimization: A Basic Course. *Kluwer Academic*. <https://doi.org/10.1007/978-1-4419-8853-9>

## Cite this entry

```bibtex
@incollection{dictml_linreg,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {linear regression},
  booktitle = {Dictionary of Applied Machine Learning},
  publisher = {Springer Nature Singapore},
  year = {2026},
  doi = {10.1007/978-981-92-2012-0_13-1},
  isbn = {978-981-92-2012-0},
  url = {https://doi.org/10.1007/978-981-92-2012-0_13-1}
}
```


---

# large language model (LLM)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/llm.html) · [typeset PDF](https://dictionaryofml.org/terms/llm.pdf) · [LaTeX](https://dictionaryofml.org/terms/llm.tex) · [Python demo](https://dictionaryofml.org/terms/llm.py) · [notebook](https://dictionaryofml.org/terms/llm.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A large language model (LLM) is an artificial neural network, typically with billions of model parameters, that implements a hypothesis map from a prompt to a response. The training of an LLM often proceeds in stages: pretraining uses self-supervised learning, splitting raw text into overlapping fragments that yield data points without manual annotation, and fine-tuning uses a small dataset of manually curated prompt–response pairs. LLM agents extend LLMs beyond text: a program interprets the response, for example as code to run or as a tool to execute, and can append the result to the next prompt, so the LLM chooses actions that the surrounding program carries out.

## Definition

Consider a chat-based artificial intelligence system such as ChatGPT, Claude, Gemini, DeepSeek, or Mistral: a user types a question, the prompt, and receives a reply, the response. The system that produces the reply is an LLM: an artificial neural network, typically a transformer (Vaswani et al., 2017) with billions of model parameters (Brown et al., 2020), trained to produce output text given input text. Each data point of this learning task is a pair of texts, and the trained LLM implements a hypothesis map from the prompt to the response (Fig. 1). A prompt and a response are sequences of tokens, so the feature space and the label space of this learning task are sets of token sequences. In contrast, a text classifier that maps a document to one of a fixed set of labels is not an LLM: it returns a label, not text.

![Figure 1 of the entry llm](https://dictionaryofml.org/terms/fig/llm-fig1.svg)

*Figure 1: An LLM as a hypothesis map from prompts to responses. Each filled circle is one data point, a pair of input text and output text; for one of the three, the prompt and the response are spelled out. The dashed curve depicts the hypothesis $\widehat{h}$ that the trained LLM implements: it assigns a response to any prompt. The axes are schematic, as the underlying feature space and label space consist of token sequences for which there is no natural ordering*

Most LLMs implement $\widehat{h}$ autoregressively: the response is generated one token at a time, each drawn from a probability distribution over the vocabulary that the network computes from the prompt and all tokens generated so far (Bengio et al., 2003; Brown et al., 2020). Alternative constructions exist: masked probabilistic models predict removed tokens and serve text analysis rather than generation (Devlin et al., 2019), and diffusion language models generate many tokens in parallel (Li et al., 2022).

The training of an LLM consists of several stages. One stage typically uses self-supervised learning: a given text is split into overlapping fragments, and each fragment amounts to a data point whose label is one of its tokens and whose features are the remaining tokens. Training sets built from text fragments require no manual annotation; a next-token fragment construction is used by Brown et al. (2020) and a masked-token one by Devlin et al. (2019). Pretraining then minimizes the average loss over the fragments, i.e., it is empirical risk minimization, the aim being a small expected loss on text fragments outside the training set. Another stage is fine-tuning, which uses a small dataset of manually curated prompt–response pairs (Ouyang et al., 2022). The pretrained model parameters are what fine-tuning adapts to each task; an LLM whose model parameters are reused across many learning tasks in this way is a foundation model (Bommasani et al., 2021).

LLM inference, i.e., computing the response for a given prompt, is computationally much cheaper than its training (Kaplan et al., 2020, Sect. 2.1). Still, the inference itself is computationally costly and is often implemented using cloud computing services. An LLM can also run locally: open-weight LLMs, whose model parameters are published (Touvron et al., 2023), can be compressed by quantization (Dettmers et al., 2022, Sect. 1). Quantization stores each model parameter in a few bits instead of a $16$-bit floating-point number. An LLM with billions of model parameters then runs on a laptop or phone (see edge computing).

The output of an LLM by itself is nothing but a sequence of letters that constitute the response. An LLM (or coding) agent interprets the response of an LLM as instructions for computations to be executed (Yao et al., 2023). For example, the prompt can instruct the LLM to produce a response that is valid source code in a programming language such as Python, as in the spelled-out data point of Fig. 1. The delivered source code can then be executed, and the result can be appended to the next prompt. The response can likewise be read as a call to an external tool, so that the LLM generates the text that specifies the actions the surrounding program carries out.

**See also:** token, vocabulary, [transformer](https://dictionaryofml.org/terms/transformer.html), [attention](https://dictionaryofml.org/terms/attention.html), [artificial neural network](https://dictionaryofml.org/terms/ann.html), word embedding, softmax function, natural language processing, foundation model, [pretraining](https://dictionaryofml.org/terms/pretraining.html), [fine-tuning](https://dictionaryofml.org/terms/finetuning.html), agent, cloud computing, edge computing.

## References

- Vaswani et al. (2017). Attention is all you need. *Adv. Neural Inf. Process. Syst.*. <https://papers.nips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html>
- Brown et al. (2020). Language Models are Few-Shot Learners. *Adv. Neural Inf. Process. Syst.*. <https://proceedings.neurips.cc/paper_files/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html>
- Bengio et al. (2003). A Neural Probabilistic Language Model. *J. Mach. Learn. Res.*. <https://www.jmlr.org/papers/v3/bengio03a.html>
- Devlin et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. *Proc. 2019 Conf. North American Chapter Assoc. Computational Linguistics: Human Lang. Technologies, Volume 1 (Long and Short Papers)*. <https://doi.org/10.18653/v1/N19-1423>
- Li et al. (2022). Diffusion-LM Improves Controllable Text Generation. *Adv. Neural Inf. Process. Syst. (NeurIPS)*. <https://proceedings.neurips.cc/paper_files/paper/2022/hash/1be5bc25d50895ee656b8c2d9eb89d6a-Abstract.html>
- Ouyang et al. (2022). Training Language Models to Follow Instructions with Human Feedback. *Adv. Neural Inf. Process. Syst. (NeurIPS)*. <https://proceedings.neurips.cc/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract.html>
- Bommasani et al. (2021). On the Opportunities and Risks of Foundation Models. <https://doi.org/10.48550/arXiv.2108.07258>
- Kaplan et al. (2020). Scaling Laws for Neural Language Models. *arXiv preprint arXiv:2001.08361*.
- Touvron et al. (2023). LLaMA: Open and Efficient Foundation Language Models. <https://arxiv.org/abs/2302.13971>
- Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. *Adv. Neural Inf. Process. Syst. (NeurIPS)*. <https://proceedings.neurips.cc/paper_files/paper/2022/hash/c3ba4962c05c49636d4c6206a97e9c8a-Abstract.html>
- Yao et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. *Int. Conf. Learn. Represent. (ICLR)*. <https://openreview.net/forum?id=WE_vluYUL-X>

## Cite this entry

```bibtex
@misc{dictml_llm,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {large language model (LLM)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/llm.html}
}
```


---

# logistic loss

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-11. [HTML](https://dictionaryofml.org/terms/logloss.html) · [typeset PDF](https://dictionaryofml.org/terms/logloss.pdf) · [LaTeX](https://dictionaryofml.org/terms/logloss.tex) · [Python demo](https://dictionaryofml.org/terms/logloss.py) · [notebook](https://dictionaryofml.org/terms/logloss.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

The logistic loss is a convex and smooth surrogate for the 0/1 loss in binary classification. Like the hinge loss, it depends on a data point only through the location of that data point relative to the decision boundary. Unlike the hinge loss it is differentiable everywhere, which suits it to gradient-based optimization methods such as gradient descent. Running empirical risk minimization with a linear model and the logistic loss gives logistic regression. The logistic loss is the negative logarithm of the conditional probability distribution of the label given the feature vector, so minimizing it is maximum likelihood estimation of the model parameters. A probabilistic model in which the features are jointly Gaussian RVs given the label induces exactly that conditional probability distribution, and it is not the only one that does.

## Definition

A spam filter is a binary classification method that learns a hypothesis that maps the features of an email to either *spam* or *no spam*. The performance of such a spam filter can be measured by counting the fraction of emails that are misclassified. This fraction is the average 0/1 loss. However, this average loss is flat almost everywhere as a function of the model parameters, so it gives gradient descent no direction to move in. Its shape is the step drawn in Fig. 1: flat on either side of the decision boundary and with no slope anywhere to follow. Practical classification methods therefore use a convex and smooth surrogate for the 0/1 loss, which is less flat. One such surrogate is the logistic loss, defined next.

Consider a binary classification problem with feature space $\mathcal{X}$ and a real-valued hypothesis $h: \mathcal{X} \rightarrow \mathbb{R}$. The two classes are encoded below by the label $y \in \{-1, +1\}$. The logistic loss incurred by $h$ on a data point $({\mathbf x}, y)$ is (Bishop, 2006)

\begin{equation}
\tag{1}
L\left(({\mathbf x}, y),h \right)
:= \ln\big(1 + \exp(-y \cdot
h({\mathbf x}))\big) \text{.}
\end{equation}

Like the hinge loss, the logistic loss Eq. (1) depends on the data point only via the margin $y \cdot h({\mathbf x})$. Normalizing the logistic loss by $\ln 2$ makes it a convex upper bound on the 0/1 loss, with equality at margin $0$ (see Fig. 1). The rescaling is what makes it a bound: at margin $0$ the logistic loss itself is $\ln 2 \approx 0.69$, below the value $1$ of the 0/1 loss there. Scaling a surrogate so that it passes through $1$ at margin $0$ is the usual convention when such loss functions are compared (Hastie et al., 2009, Sect. 10.6).

![Figure 1 of the entry logloss](https://dictionaryofml.org/terms/fig/logloss-fig1.svg)

*Figure 1: The logistic loss $L\left(({\mathbf x}, y),h \right)$ divided by $\ln 2$ (solid curve) and the 0/1 loss (dashed curve) as functions of the margin $y \cdot h({\mathbf x})$. The rescaled logistic loss lies on or above the 0/1 loss for every margin, with equality at margin $0$; the rescaling is needed because $\ln\big(1 + \exp(0)\big) = \ln 2 < 1$*

Unlike the hinge loss, the logistic loss is everywhere differentiable. It is therefore well suited for gradient-based optimization methods such as gradient descent. Empirical risk minimization with a linear model and the logistic loss yields logistic regression.

The logistic loss arises naturally from a simple probabilistic model in which the data points are independent and identically distributed RVs with a common probability distribution $\mathbb{P}^{({\mathbf x}, y)}$. Conditioned on the label $y$, the feature vector ${\mathbf x}$ is a Gaussian random variable with mean vector ${\boldsymbol \mu}^{(-1)}$ for $y = -1$ and ${\boldsymbol \mu}^{(+1)}$ for $y = +1$. The conditional covariance matrix is the same for both, and fixed to $\mathbf{I}$.

Under that probabilistic model, the posterior probability of $y = +1$ is the sigmoid function $\sigma(h({\mathbf x}))$ with $\sigma(z) = 1/(1 + e^{-z})$. The identity holds for the affine hypothesis $h({\mathbf x}) = {\mathbf w}^{\top} {\mathbf x}
+ b$ with the model parameters (Bishop, 2006, Sect. 4.2.1, Eqs. (4.65)–(4.67))

$$
{\mathbf w} = {\boldsymbol \mu}^{(+1)} - {\boldsymbol \mu}^{(-1)} \text{,} \quad
b = - \tfrac{1}{2} \big(
\left\lVert {\boldsymbol \mu}^{(+1)} \right\rVert_{2}^{2}
- \left\lVert {\boldsymbol \mu}^{(-1)} \right\rVert_{2}^{2} \big)
+ \ln \frac{\mathbb{P}\left(y = +1\right)}
{\mathbb{P}\left(y = -1\right)} \text{.}
$$

It is the shared covariance matrix that makes $h$ affine: the terms quadratic in ${\mathbf x}$ are the same under both labels and cancel in the ratio, leaving only the inner product with ${\mathbf w}$. When the two labels are equally likely the logarithm vanishes and $b$ is the midpoint term alone. The probability of $y = -1$ is then $\sigma(-h({\mathbf x}))$, because the sigmoid function satisfies $\sigma(-z) = 1 - \sigma(z)$ (Bishop, 2006, Eq. 4.60). The logistic loss is thus the negative logarithm of the conditional probability of the label given the feature vector, and minimizing the average logistic loss is maximum likelihood estimation of the weight vector ${\mathbf w}$ and the offset $b$ of $h$ under that conditional probability distribution (Bishop, 2006, Sect. 4.3.2; Murphy, 2012, Sect. 8.3). The Gaussian probabilistic model is not the only one that induces this conditional probability distribution. If, given the label, the feature vector has a density of the form $h({\mathbf x}) \exp\big( ({\mathbf w}^{(y)})^{\top}
\mathbf{t}({\mathbf x}) - A({\mathbf w}^{(y)}) \big)$, a member of an exponential family whose natural parameter ${\mathbf w}^{(y)}$ depends on the label while the base measure $h$ and the sufficient statistic $\mathbf{t}({\mathbf x})$ do not, then the logarithm of the posterior odds of $y = +1$ is affine in $\mathbf{t}({\mathbf x})$, and the posterior is again the sigmoid function of an affine function (Bishop, 2006, Sect. 4.2.4). Its weight vector is the difference of the two natural parameters, ${\mathbf w} = {\mathbf w}^{(+1)} - {\mathbf w}^{(-1)}$, which for the Gaussian model above, where $\mathbf{t}({\mathbf x}) = {\mathbf x}$ and the natural parameter is the mean vector, is the identity ${\mathbf w} = {\boldsymbol \mu}^{(+1)} - {\boldsymbol \mu}^{(-1)}$ displayed there. Conditionally independent binary features, the model behind naive Bayes, are another case (Bishop, 2006, Sect. 4.2.3).

A binary classification problem whose label space is some other pair $\{a, b\}$ is covered by an encoding $e: \{a, b\} \rightarrow \{-1, +1\}$ with $e(a) = -1$ and $e(b) = +1$. The logistic loss Eq. (1) is then evaluated at the encoded label,

$$
L\left(({\mathbf x}, y),h \right)
= \ln\big(1 + \exp(-e(y) \cdot
h({\mathbf x}))\big) \text{.}
$$

The label spaces $\{0, 1\}$ and $\{\textit{no spam}, \textit{spam}\}$ are both handled this way. Take the label space $\{0, 1\}$, with $e(0) = -1$ and $e(1) = +1$. The loss can then be written out as

$$
- y \ln \sigma\big(h({\mathbf x})\big)
- (1 - y) \ln \big(1 -
\sigma(h({\mathbf x}))\big) \text{,}
$$

which is the binary cross-entropy (BCE) loss: the cross-entropy between the label and the predicted probability $\sigma(h({\mathbf x}))$, with the label $1$ leaving the second summand zero and the label $0$ the first. Summed over a training set it is the cross-entropy error function of logistic regression (Bishop, 2006, Eq. 4.90).

Written that way the loss no longer needs the label to be one of two values. Let $\tilde{y}$ be a soft label, a number between $0$ and $1$ giving the probability that the data point belongs to the class encoded by $1$. Putting $\tilde{y}$ in place of $y$ gives the cross-entropy loss

$$
- \tilde{y} \ln \sigma\big(h({\mathbf x})\big)
- (1 - \tilde{y}) \ln \big(1 -
\sigma(h({\mathbf x}))\big) \text{,}
$$

which reduces to the previous display at $\tilde{y} \in \{0, 1\}$ and carries over to more than two classes through the softmax function.

The value of the logistic loss is harder to interpret than the 0/1 loss, because it depends on the size of the margin and not only on its sign. Each hypothesis compared below is a linear classifier $h({\mathbf x}) = {\mathbf w}^{\top} {\mathbf x}
+ b$. Take the three data points of Fig. 2, two of them with label $+1$ at $(1,1)$ and $(2,3)$ and one with label $-1$ at $(3,1)$, and the linear classifier $\widehat{h}^{(1)}({\mathbf x}) = -0.1 x_{1}
+ 0.05 x_{2} + 0.15$. Its margins are $0.1$ on each of the three, so it classifies all three correctly and incurs an average logistic loss of $0.644$. Multiplying its model parameters by $50$ gives $50 \widehat{h}^{(1)}$, which leaves the decision boundary exactly where it was and so changes no prediction, yet the average logistic loss falls to $0.0067$, a factor of $96$. Two hypotheses far apart in logistic loss can therefore make exactly the same decisions.

The logistic loss and the 0/1 loss can also rank two hypotheses in opposite orders. A second linear classifier $\widehat{h}^{(2)}$ has a decision boundary parallel to the first but shifted, which leaves the data point at $(3,1)$ on the wrong side. Its margins are $5$, $5$, and $-0.5$, an accuracy of $2/3$ against $1$ for $\widehat{h}^{(1)}$, and it still attains the smaller average logistic loss, $0.329$ against $0.644$. Its two large margins cost $0.0067$ each, so even the single negative margin, at $0.974$, totals less than the three small margins of $\widehat{h}^{(1)}$ at $0.644$ each. Choosing between the two by logistic loss alone selects the less accurate one. Only one reading of the number is safe, the upper bound above: a small logistic loss forces a small 0/1 loss.

![Figure 2 of the entry logloss](https://dictionaryofml.org/terms/fig/logloss-fig2.svg)

*Figure 2: The three data points of the example, the two with label $y = +1$ drawn as filled circles and the one with label $y = -1$ as an open square, and the decision boundaries of the hypotheses compared on them. Scaling $\widehat{h}^{(1)}$ by $50$ multiplies its model parameters by $50$ and so leaves the decision boundary exactly where it was (solid line): every prediction is unchanged while the average logistic loss falls. The decision boundary of $\widehat{h}^{(2)}$ (dashed line) is parallel but shifted, which puts the data point at $(3,1)$ on the wrong side. Data generated by pythondemos/logloss.py*

**Synonyms:** log loss, binary cross-entropy (BCE) loss.

**See also:** $\bf 0/1$ loss, hinge loss, margin, [accuracy](https://dictionaryofml.org/terms/accuracy.html), cross-entropy, probabilistic model, [sigmoid function](https://dictionaryofml.org/terms/sigmoid.html), classifier, [logistic regression](https://dictionaryofml.org/terms/logreg.html), [gradient descent](https://dictionaryofml.org/terms/gd.html), [convex](https://dictionaryofml.org/terms/convex.html), [binary classification](https://dictionaryofml.org/terms/binclass.html).

## References

- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Murphy (2012). Machine Learning: A Probabilistic Perspective. *MIT Press*.

## Cite this entry

```bibtex
@misc{dictml_logloss,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {logistic loss},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/logloss.html}
}
```


---

# logistic regression

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-11. [HTML](https://dictionaryofml.org/terms/logreg.html) · [typeset PDF](https://dictionaryofml.org/terms/logreg.pdf) · [LaTeX](https://dictionaryofml.org/terms/logreg.tex) · [Python demo](https://dictionaryofml.org/terms/logreg.py) · [notebook](https://dictionaryofml.org/terms/logreg.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Logistic regression learns a linear hypothesis map for binary classification, that is, for data points whose label takes one of two values, $y \in \{-1, 1\}$. The predicted label is the sign of $h({\mathbf x})$, and the absolute value $|h({\mathbf x})|$ measures the confidence in the classification. The parameters are learned by empirical risk minimization with the average logistic loss on the training set. This is equivalent to maximum likelihood estimation under a simple probabilistic model, under which $\sigma(h({\mathbf x}))$ is the probability of the label $1$. When a data point carries this probability as a soft label instead of a class, the logistic loss is replaced by the cross-entropy between soft label and prediction, equivalently their Kullback–Leibler divergence. The resulting objective function is convex and smooth, so variants of gradient descent can be used to minimize it.

## Definition

Consider predicting whether it freezes tomorrow from recent temperature measurements. Records from a weather station turn this question into data: each day is a data point with two numeric features, the maximum temperature of the day and that of the preceding day, collected in the feature vector ${\mathbf x} = (x_{1}, x_{2})^{\top} \in
\mathbb{R}^{2}$. The label $y \in \{-1, 1\}$ is $y = 1$ if the minimum temperature of the following day stays at or above $0^{\circ}$C and $y = -1$ otherwise.

Fig. 1 shows such data points. Logistic regression learns, for such a binary classification task, a linear hypothesis map $h({\mathbf x}) = {\mathbf w}^{\top} {\mathbf x} + b$ with weight vector ${\mathbf w}$ and offset $b$ (Bishop, 2006; Hastie et al., 2009). Despite its name, the method solves a classification problem: the regression refers to fitting the real-valued map $h$, not to a numeric label. The set of feature vectors with $h({\mathbf x}) = 0$ is the decision boundary between the two classes, and ${\mathbf w}$ is its normal vector.

![Figure 1 of the entry logreg](https://dictionaryofml.org/terms/fig/logreg-fig1.svg)

*Figure 1: Data points for the frost forecast at the weather station Krems an der Donau: each day is placed at the maximum temperature of the day ($x_{1}$) and of the preceding day ($x_{2}$). Days followed by a freezing day (minimum temperature below $0^{\circ}$C) are drawn as open triangles; they concentrate where both features are small, but the two classes overlap. Twenty of the 364 days are drawn, every 22nd day without and every 12th day with a freezing successor. The dashed line is the decision boundary of the hypothesis learned from all 364 days and the solid arrow its normal vector ${\mathbf w}$. Data generated by pythondemos/logreg.py*

The prediction, i.e., the classification, is obtained from the sign of $h({\mathbf x})$, which makes the learned $h$ a linear classifier whose decision boundary is the hyperplane ${\mathbf w}^{\top} {\mathbf x} + b = 0$ with normal vector ${\mathbf w}$. The absolute value $|h({\mathbf x})|$ is used as a measure of the confidence in the classification: it grows with the distance of ${\mathbf x}$ from the decision boundary, and a value close to zero indicates low confidence.

The parameters ${\mathbf w}$ and $b$ of logistic regression are learned via empirical risk minimization. The loss it minimizes is the logistic loss. On a data point $({\mathbf x}, y)$, a hypothesis $h$ incurs

$$
L\left(({\mathbf x}, y),h \right) =
\ln\big(1 + \exp(-y \, h({\mathbf x}))\big)
\text{.}
$$

Fig. 2 shows this loss for a data point with $y = 1$ as a function of $h({\mathbf x})$. A correct classification with high confidence incurs a small loss. A wrong classification incurs a loss whose size grows with the confidence in the wrong classification.

![Figure 2 of the entry logreg](https://dictionaryofml.org/terms/fig/logreg-fig2.svg)

*Figure 2: The logistic loss incurred on a data point with label $y = 1$ as a function of the hypothesis value $h({\mathbf x})$. A correct classification, $h({\mathbf x}) > 0$, with high confidence incurs a small loss; the loss grows as $h({\mathbf x})$ becomes negative, that is, as the confidence in a wrong classification increases. The dashed line marks the decision boundary $h({\mathbf x}) = 0$, and the arrows point toward growing confidence in a correct (right) or wrong (left) classification. Data generated by pythondemos/logreg.py*

Averaging this loss over the training set $\mathcal{D}^{(\rm train)} =
\{({\mathbf x}^{(r)},
y^{(r)})\}_{r=1}^{m}$ of $m$ data points, empirical risk minimization delivers

$$
\begin{aligned}
(\widehat{{\mathbf w}}, \widehat{b}) &\in \mathop{\mathrm{arg\,min}}_{{\mathbf w} \in
\mathbb{R}^{d},\, b \in \mathbb{R}} f({\mathbf w}, b), \quad\text{with}\\
f({\mathbf w}, b) &:= \frac{1}{m}
\sum_{r=1}^{m}
\ln\Big(1 + \exp\big(-y^{(r)}
({\mathbf w}^{\top} {\mathbf x}^{(r)} + b)\big)\Big) \text{.}
\end{aligned}
$$

The logistic loss is the negative log-likelihood of the label under a simple probabilistic model for the classification task.

In this probabilistic model, given the label $y$, the feature vector ${\mathbf x}$ is a Gaussian random variable whose mean ${\boldsymbol \mu}^{(y)}$ depends on the label while its covariance matrix ${\mathbf C}$ does not; the two means ${\boldsymbol \mu}^{(-1)}, {\boldsymbol \mu}^{(+1)} \in
\mathbb{R}^{d}$ are the parameters of the model. When both labels are equally probable a priori, the best linear classifier under this model assigns ${\mathbf x}$ to the label whose mean is nearer, with distances measured by ${\mathbf C}^{-1}$, and its decision boundary is the hyperplane through the midpoint of the two means (Bishop, 2006, Sect. 4.2.1). Its weight vector is ${\mathbf w} = {\mathbf C}^{-1}\big({\boldsymbol \mu}^{(+1)} - {\boldsymbol \mu}^{(-1)}\big)$. The factor $\tfrac{1}{2}$ that puts the decision boundary through the midpoint belongs to the offset $b$ and not to ${\mathbf w}$, as does the logarithm of the ratio of the two prior probabilities when they differ: any positive multiple of ${\mathbf w}$ leaves the decision boundary where it is, but only this one reproduces the posterior probability below. Under the model, the posterior probability of $y = 1$ is the sigmoid function $\sigma({\mathbf w}^{\top}
{\mathbf x} + b)$ with $\sigma(z) = 1 / (1 + e^{-z})$ (Bishop, 2006, Sect. 4.2.1, Eqs. (4.65)–(4.67)). Logistic regression estimates ${\mathbf w}$ and $b$ directly, by maximizing the conditional likelihood function of the labels given the feature vectors. This leaves the distribution of the feature vectors unspecified and is a different estimator from maximum likelihood for the means and the covariance matrix of the Gaussian model (Hastie et al., 2009, Sect. 4.4.5). Maximizing the conditional likelihood function coincides with minimizing the average logistic loss $f$ (Murphy, 2012, Sect. 8.3).

The probabilistic model makes the confidence reading of $|h({\mathbf x})|$ above precise: $\sigma(h({\mathbf x}))$ is the probability that the label of a data point with feature vector ${\mathbf x}$ is $y = 1$, and $\sigma(-h({\mathbf x})) = 1 - \sigma(h({\mathbf x}))$ the probability of $y = -1$. A forecast $h({\mathbf x}) = 2$ thus says that $88$ percent of the days with these temperatures are followed by a frost-free day, and a forecast of $0$ that half of them are. Whether these numbers can be trusted is the question of calibration (Dawid, 1982).

In most applications a data point carries only its label, the class that occurred. The logistic loss is the loss for this case. It is the cross-entropy between the probability mass function concentrated on the observed label and the predicted probability mass function $(\sigma(h({\mathbf x})), 1 -
\sigma(h({\mathbf x})))$ (Bishop, 2006, Sect. 4.3.2). Some data points carry more than a class: a soft label $\tilde{y} \in [0, 1]$, the probability of $y = 1$ itself. It may be the frequency of frost among the recorded days with the same temperatures, the share of several annotators who chose the class, or the output of a larger classifier, as in knowledge distillation. For such a data point the logistic loss is replaced by the cross-entropy between the soft label and the prediction,

$$
L\left(({\mathbf x}, \tilde{y}),h \right) =
- \tilde{y} \ln \sigma(h({\mathbf x}))
- (1 - \tilde{y}) \ln \big(1 - \sigma(h({\mathbf x}))\big)
\text{,}
$$

which for $\tilde{y} \in \{0, 1\}$ is the logistic loss again. Learning then minimizes the average of this loss over the training set, as before.

This cross-entropy equals the Kullback–Leibler divergence from the predicted probability mass function to the soft label probability mass function $(\tilde{y}, 1
- \tilde{y})$ plus the entropy of the soft label probability mass function, which does not depend on $h$ (Bishop, 2006, Sect. 1.6.1, Eq. (1.113)). Minimizing either therefore learns the same $h$, and the loss on a data point is smallest exactly when $\sigma(h({\mathbf x})) = \tilde{y}$, when the confidence of the prediction matches the probability of the class. A hard label can never be matched this way, since $\sigma$ never reaches $0$ or $1$. This is why the logistic loss keeps decreasing as $|h({\mathbf x})|$ grows on a correctly classified data point.

With hard or soft labels, the parameters are found by minimizing the average loss $f$. Unlike linear regression with the squared error loss, the minimization admits no closed-form solution (Bishop, 2006, Sect. 4.3.3). As in linear regression, however, the objective function $f$ is convex (Murphy, 2012, Sect. 8.3.1, Eq. (8.7)) and smooth, so gradient descent applies. The parameters are collected as the pair $({\mathbf w}, b)$, and the gradient descent step reads

$$
({\mathbf w}, b)^{(t+1)} = T\big(({\mathbf w}, b)^{(t)}\big)
\text{, with } T({\mathbf w}, b) := ({\mathbf w}, b) - \eta
\nabla f({\mathbf w}, b) \text{, for } t = 0,1,\ldots \text{,}
$$

with learning rate $\eta$. The gradient $\nabla f$ of the average logistic loss has the weight part

$$
\nabla_{{\mathbf w}} f = -\frac{1}{m}
\sum_{r=1}^{m} y^{(r)}
{\mathbf x}^{(r)} \,
\sigma\big(-y^{(r)} ({\mathbf w}^{\top}
{\mathbf x}^{(r)} + b)\big) \text{,}
$$

and analogously for $b$.

The pair $({\mathbf w}, b)$ is a fixed point of the operator $T$ precisely when $\nabla f({\mathbf w}, b) = \mathbf{0}$, which by convexity of $f$ is exactly when $({\mathbf w}, b)$ is a minimizer. It can be shown that $\nabla f$ is Lipschitz continuous with some constant $L$ (Hastie et al., 2009). For a constant learning rate $\eta < 2/L$, every gradient descent step decreases $f$ (Beck, 2017, Lemma 10.4). For the standard choice $\eta = 1/L$ and a minimizer $(\widehat{{\mathbf w}}, \widehat{b})$ of $f$, the gap $f\big(({\mathbf w}, b)^{(t)}\big) - \min f$ at the $t$th iterate is at most $2 L
\left\Vert {({\mathbf w}, b)^{(1)} - (\widehat{{\mathbf w}}, \widehat{b})} \right\Vert^{2}_{2}
/ (t - 1)$ (Bubeck, 2015, Thm. 3.3). The iterates $({\mathbf w}, b)^{(t)}$ themselves need not converge: on a linearly separable training set the minimum is not attained (Hastie et al., 2009, Sect. 4.4.5), and the norm of the iterates grows without bound while the training error tends to zero. Adding the penalty term $\alpha \left\Vert {{\mathbf w}} \right\Vert^{2}_{2}$, with a positive $\alpha$, to $f$ removes this singularity (Bishop, 2006, Sect. 4.3.2). The penalized objective function is strictly convex in $({\mathbf w}, b)$: its second derivative along a direction $({\mathbf u}, c) \neq (\mathbf{0}, 0)$ is

$$
\frac{1}{m} \sum_{r=1}^{m}
s^{(r)} \big({\mathbf u}^{\top}{\mathbf x}^{(r)} + c\big)^{2}
+ 2\alpha \left\Vert {{\mathbf u}} \right\Vert^{2}_{2} \text{,}
$$

with $s^{(r)} = \sigma\big(h({\mathbf x}^{(r)})\big)\,
\sigma\big(-h({\mathbf x}^{(r)})\big) > 0$, and this vanishes only for ${\mathbf u} = \mathbf{0}$ and $c = 0$. When both labels occur in the training set, the penalized objective function also grows without bound along every unbounded sequence of pairs $({\mathbf w}, b)$, so it has a unique minimizer. Its gradient is Lipschitz continuous with constant $L + 2\alpha$. For a learning rate $\eta < 2/(L + 2\alpha)$ the values of the gradient descent iterates again converge to the minimum, the iterates stay bounded, and they approach the unique minimizer.

In practice, plain gradient descent is seldom used to minimize $f$: the smooth convex objective function is minimized faster by Newton's method, equivalently iteratively reweighted least squares (Hastie et al., 2009, Sect. 4.4.1, Eq. (4.28)), by the quasi-Newton solver L-BFGS (Liu and Nocedal, 1989), and, for a large training set, by coordinate descent (Wright, 2015).

What these solvers minimize is the average logistic loss, but that is not the only loss with which a linear classifier can be learned. In principle, the squared error loss serves too: with the label values encoded as the real numbers $+1$ and $-1$, the squared error $\big(y - ({\mathbf w}^{\top}{\mathbf x} + b)\big)^{2}$ is well defined. Minimizing the squared error loss, however, has a flaw: it penalizes a prediction ${\mathbf w}^{\top}{\mathbf x} + b$ that is large, positive, and correct, while the logistic loss keeps decreasing as this margin grows (Fig. 3). The hinge loss of the support vector machine shares this behavior on a correctly classified data point but stays flat once the margin exceeds one.

![Figure 3 of the entry logreg](https://dictionaryofml.org/terms/fig/logreg-fig3.svg)

*Figure 3: The logistic loss, squared error loss and hinge loss incurred on a data point with label $y = 1$ as a function of the hypothesis value $h({\mathbf x})$. The squared error loss turns upward and penalizes a confident correct prediction, whereas the logistic loss keeps decreasing; the hinge loss stays flat once the margin exceeds one. The dashed line marks the decision boundary $h({\mathbf x}) = 0$, and the arrows point toward growing confidence in a correct (right) or wrong (left) classification. Data generated by pythondemos/logreg.py*

For more than two label values, logistic regression extends in two ways. A simple reduction trains one binary classification logistic regression per label, each separating that label from the rest, and predicts with the most confident of them; this is one-vs-rest classification. An alternative replaces the sigmoid function by the softmax function, which models the probabilities of all labels jointly, so that they sum to one and no label is left without one; fitting it by maximum likelihood yields multinomial logistic regression (Bishop, 2006, Sect. 4.3.4).

**See also:** [classification](https://dictionaryofml.org/terms/classification.html), [binary classification](https://dictionaryofml.org/terms/binclass.html), [linear classifier](https://dictionaryofml.org/terms/linclass.html), [decision boundary](https://dictionaryofml.org/terms/decisionboundary.html), [sigmoid function](https://dictionaryofml.org/terms/sigmoid.html), [logistic loss](https://dictionaryofml.org/terms/logloss.html), squared error loss, hinge loss, [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), [gradient descent](https://dictionaryofml.org/terms/gd.html), maximum likelihood, [linear regression](https://dictionaryofml.org/terms/linreg.html), softmax function, [regularization](https://dictionaryofml.org/terms/regularization.html), calibration, soft label, cross-entropy, Kullback–Leibler divergence.

## References

- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Murphy (2012). Machine Learning: A Probabilistic Perspective. *MIT Press*.
- Dawid (1982). The Well-Calibrated Bayesian. *Journal of the American Statistical Association*.
- Beck (2017). First-Order Methods in Optimization. *SIAM-Society for Industrial and Applied Mathematics*.
- Bubeck (2015). Convex Optimization: Algorithms and Complexity. *Found. Trends Mach. Learn.*. <https://doi.org/10.1561/2200000050>
- Liu and Nocedal (1989). On the Limited Memory Method for Large Scale Optimization. *Mathematical Programming B*.
- Wright (2015). Coordinate Descent Algorithms. *Mathematical Programming*. <https://doi.org/10.1007/s10107-015-0892-3>

## Cite this entry

```bibtex
@misc{dictml_logreg,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {logistic regression},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/logreg.html}
}
```


---

# loss

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-10-29. [HTML](https://dictionaryofml.org/terms/loss.html) · [typeset PDF](https://dictionaryofml.org/terms/loss.pdf) · [LaTeX](https://dictionaryofml.org/terms/loss.tex) · [Python demo](https://dictionaryofml.org/terms/loss.py) · [notebook](https://dictionaryofml.org/terms/loss.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

The loss incurred by a hypothesis on a single data point is a scalar that quantifies the error of the hypothesis when evaluated on that data point. The ultimate goal of machine learning methods is to train a model such that the learned hypothesis delivers predictions with minimum loss. A key characteristic of machine learning applications and methods is the accessibility of loss values. In supervised learning, the loss can be evaluated for every hypothesis. In reinforcement learning, the loss is observed only for the action actually taken.

## Definition

In temperature forecasting, the quality of a predicted temperature is naturally measured by how far it deviates from the temperature actually measured the next day. The loss formalizes this idea. The loss incurred by a hypothesis $h \in \mathcal{H}$ on a single data point is a scalar that quantifies the error of $h$ when evaluated on that data point (Shalev-Shwartz and Ben-David, 2014, Sect. 2.1). A smaller loss value indicates a smaller error. In supervised learning, this error is the discrepancy between the prediction $h({\mathbf x})$ delivered for the features ${\mathbf x}$ and the label $y$ of that data point. Some losses (e.g., the squared error loss) take a continuum of values, while others (e.g., the 0/1 loss) are binary, indicating only whether the prediction matches the label. For the temperature forecast, the loss on a given day can be the squared difference between the predicted temperature (the prediction) and the measured one (the label); see Fig. 1.

![Figure 1 of the entry loss](https://dictionaryofml.org/terms/fig/loss-fig1.svg)

*Figure 1: Loss on a single data point representing one day. The true label is the measured maximum daytime temperature, $y = 12\,^{\circ}\mathrm{C}$; the hypothesis predicts $h({\mathbf x}) = 8\,^{\circ}\mathrm{C}$. Under the squared error loss, the loss incurred on this data point is $\big(y - h({\mathbf x})\big)^{2} = 16\,(\,^{\circ}\mathrm{C})^{2}$*

Loss values are typically nonnegative: many losses are built from a norm or metric and are therefore bounded below by zero. Nonnegativity is not required, however: the logarithmic loss (negative log-likelihood) underlying maximum likelihood methods can take negative values, since a probability density function can exceed one. The loss is also not specific to supervised learning. In unsupervised learning there is no label: a clustering loss measures how far a data point lies from its assigned cluster centroid, and the loss of an autoencoder is the reconstruction error between a data point and its reconstruction.

Machine learning methods use observed loss values to construct an objective function for model training. The average loss across a training set of data points is the empirical risk that empirical risk minimization minimizes.

Machine learning applications can be categorized by how the loss values are accessible. In supervised learning, the loss can, in principle, be evaluated for every hypothesis in the hypothesis space, since the true label of each training data point is observed.

In reinforcement learning, the situation differs (Sutton and Barto, 2018). At each time step, the machine learning method (implemented by an agent) selects an action via the currently used hypothesis. The agent then observes the loss incurred by the action actually taken. The loss that would have been incurred by the other actions is not revealed. This limited accessibility of loss values is known as bandit (or partial) feedback (Hazan, 2022, Sect. 6.1). For example, a route-planning system learns the actual travel time only for the route the driver took, not for any alternative route. Similarly, a self-driving car selects a steering direction, and the incurred loss can be measured from onboard sensors such as collision detectors and lane-departure warnings, but only for the trajectory actually driven.

**Synonyms:** cost.

**See also:** loss function, empirical risk, [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), [data point](https://dictionaryofml.org/terms/datapoint.html), [hypothesis](https://dictionaryofml.org/terms/hypothesis.html), [reinforcement learning](https://dictionaryofml.org/terms/reinforcementlearning.html), [supervised learning](https://dictionaryofml.org/terms/supervisedlearning.html).

## References

- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Sutton and Barto (2018). Reinforcement Learning: An Introduction. *MIT Press*.
- Hazan (2022). Introduction to Online Convex Optimization. *MIT Press*. <https://sites.google.com/view/intro-oco/>

## Cite this entry

```bibtex
@incollection{dictml_loss,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {loss},
  booktitle = {Dictionary of Applied Machine Learning},
  publisher = {Springer Nature Singapore},
  year = {2026},
  doi = {10.1007/978-981-92-2012-0_3-1},
  isbn = {978-981-92-2012-0},
  url = {https://doi.org/10.1007/978-981-92-2012-0_3-1}
}
```


---

# matrix

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/matrix.html) · [typeset PDF](https://dictionaryofml.org/terms/matrix.pdf) · [LaTeX](https://dictionaryofml.org/terms/matrix.tex) · [Python demo](https://dictionaryofml.org/terms/matrix.py) · [notebook](https://dictionaryofml.org/terms/matrix.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A matrix is a two-dimensional array of numbers, arranged in rows and columns. The entry $A_{r,j}$ of a matrix ${\mathbf A} \in \mathbb{R}^{m \times d}$ sits in row $r$ and column $j$. A prominent example in machine learning is the feature matrix of a dataset, obtained by stacking the feature vectors of $m$ data points row-wise. A matrix represents several distinct mathematical objects. One is a system of linear equations, such as the normal equations of linear regression. Another is a linear map between two vector spaces, after fixing a basis for each. A third is a tabular dataset with one row per data point and one column per feature.

## Definition

Consider a spreadsheet of daily weather records at one station: each row holds a single day, each column one measurement such as temperature or humidity. Such a tabular dataset holds $m$ data points, each described by the same number $d$ of numerical features. The features of the $r$-th data point are collected in its feature vector ${\mathbf x}^{(r)} = \big(x^{(r)}_{1},
\ldots, x^{(r)}_{d}\big)^{\top} \in
\mathbb{R}^{d}$, one entry per column of the spreadsheet. Stacking these $m$ feature vectors row-wise, so that the $r$-th row is $\big({\mathbf x}^{(r)}\big)^{\top}$, produces an $m \times d$ rectangular array of numbers. Its entry in row $r$ and column $j$ is the $j$-th feature of the $r$-th data point. This two-dimensional numeric array is referred to as the feature matrix ${\mathbf X}$ of the dataset; Fig. 1 carries out the construction on three days of the weather example.

![Figure 1 of the entry matrix](https://dictionaryofml.org/terms/fig/matrix-fig1.svg)

*Figure 1: A tabular dataset of $m = 3$ data points and the feature matrix built from it. Each row of the table is one day, whose $d = 3$ measurements, temperature (temp.), humidity (humid.), and pressure (press.), form the feature vector ${\mathbf x}^{(r)} \in \mathbb{R}^{3}$; the day column indexes the data points and is not itself a feature, so it does not appear in ${\mathbf X}$. Stacking ${\mathbf x}^{(1)}, {\mathbf x}^{(2)}, {\mathbf x}^{(3)}$ as rows makes the entry $X_{r,j}$ the $j$-th measurement taken on day $r$: $X_{2,3} = 1008$ is the pressure on day $2$*

More generally, a matrix of size $m \times d$ is a two-dimensional array of numbers denoted by

$$
{\mathbf A} = \begin{pmatrix}
A_{1,1} & A_{1,2} & \dots & A_{1,d} \\
A_{2,1} & A_{2,2} & \dots & A_{2,d} \\
\vdots & \vdots & \ddots & \vdots \\
A_{m,1} & A_{m,2} & \dots & A_{m,d}
\end{pmatrix} \in \mathbb{R}^{m \times d}\text{.}
$$

Here, $A_{r,j}$ denotes the matrix entry in the $r$-th row and the $j$-th column. A digital image is a matrix of exactly this kind. Recording how green each pixel is turns a $16 \times 16$ image into a $16 \times 16$ matrix $\mathbf{G}$ whose two indices are pixel positions (Fig. 2).

![Figure 2 of the entry matrix](https://dictionaryofml.org/terms/fig/matrix-fig2.svg)

*Figure 2: A $16 \times 16$ pixel image of the ``for all'' quantifier $\forall$ and the matrix of its green channel. Pixel $(r,j)$ has greenness $G_{r, j} = 1$ where the glyph is drawn and $0$ elsewhere, so the $1$s trace the same shape in $\mathbf{G}$ that the shaded squares trace in the image*

Matrices represent several distinct mathematical objects (Strang, 2016), including the following:

- Systems of linear equations: A matrix collects the coefficients of a system of linear equations,

$$
\begin{pmatrix}
A_{1,1} & A_{1,2} \\
A_{2,1} & A_{2,2}
\end{pmatrix}
\begin{pmatrix}
x_1 \\
x_2
\end{pmatrix}
=\begin{pmatrix}
b_1 \\
b_2
\end{pmatrix}
\text{, written compactly as } {\mathbf A} {\mathbf x} = {\mathbf b} \text{.}
$$

Here, ${\mathbf x}$ is the vector of unknowns and ${\mathbf b}$ the right-hand side. One important example is the normal equations ${\mathbf X}^{\top}{\mathbf X}\,{\mathbf w} = {\mathbf X}^{\top}{\mathbf y}$, whose solutions are the model parameters that minimize the training error in linear regression, as the derivation following Fig. 3 shows.
- Linear maps: Consider two vector spaces $\mathcal{U}$ and $\mathcal{V}$, of dimension $d$ and $m$, respectively. Fixing a basis ${\mathbf u}^{(1)},\,\ldots,\,{\mathbf u}^{(d)}$ for $\mathcal{U}$ and a basis ${\mathbf v}^{(1)},\,\ldots,\,{\mathbf v}^{(m)}$ for $\mathcal{V}$, each matrix ${\mathbf A} \in \mathbb{R}^{m \times d}$ defines a linear map $f^{({\mathbf A})}: \mathcal{U} \rightarrow \mathcal{V}$ (see Fig. 3) via

$$
f^{({\mathbf A})}\big({\mathbf u}^{(j)}\big) = \sum_{r=1}^{m} A_{r,j} {\mathbf v}^{(r)}\text{.}
$$

Taking $\mathcal{U} = \mathbb{R}^{d}$ and $\mathcal{V} = \mathbb{R}^{m}$ with the standard bases, the image of a vector ${\mathbf x}$ under $f^{({\mathbf A})}$ is the matrix-vector product $f^{({\mathbf A})}({\mathbf x}) = {\mathbf A}{\mathbf x}$ (Axler, 2024, Thm. 3.76; Strang, 2016, Sect. 8.2).
- Datasets: As in the opening example, a matrix can represent a dataset with each row holding a single data point and each column a specific feature or label.

![Figure 3 of the entry matrix](https://dictionaryofml.org/terms/fig/matrix-fig3.svg)

*Figure 3: The matrix ${\mathbf A}$ defines a linear map $f^{({\mathbf A})}$ between two vector spaces: the basis vector ${\mathbf u}^{(j)}$ is mapped to the vector (thick red arrows) whose coordinates form the $j$-th column of ${\mathbf A}$*

Written with the feature matrix, the training error that empirical risk minimization-based linear regression minimizes becomes a succinct algebraic expression. Collecting the labels of the training set in ${\mathbf y} \in \mathbb{R}^{m}$, the training error of the hypothesis $h^{({\mathbf w})}({\mathbf x}) := {\mathbf w}^{\top}{\mathbf x}$ is

$$
(1/m)\,\left\lVert {\mathbf X} {\mathbf w} - {\mathbf y} \right\rVert_{2}^{2}
\text{,}
$$

with gradient $(2/m)\,{\mathbf X}^{\top}({\mathbf X} {\mathbf w} - {\mathbf y})$. Each gradient descent step is therefore one multiplication by ${\mathbf X}$ followed by one by ${\mathbf X}^{\top}$, and equating the gradient to zero leads back to the normal equations from the first example above.

Matrices appear across machine learning beyond the feature matrix. The Jacobian matrix and the Hessian collect the first- and second-order partial derivatives of a multivariate function and are the input of gradient-based training. The covariance matrix records the pairwise covariances of a probability distribution over features. Principal component analysis, singular value decomposition, and a random projection factor or compress the feature matrix to lower its dimension. A kernel method replaces explicit feature vectors by a matrix of pairwise inner products, and a tabular Markov decision process stores its transition probabilities as a matrix. A graph is represented by matrices as well: its adjacency matrix lists which nodes are joined and with what edge weight. The Laplacian matrix assembled from those weights (Luxburg, 2007, Sect. 3) is the matrix whose eigenvectors graph clustering uses to split the nodes into clusters.

**See also:** linear map, [dataset](https://dictionaryofml.org/terms/dataset.html), linear model, vector, [feature matrix](https://dictionaryofml.org/terms/featuremtx.html), normal equations, array, transpose.

## References

- Strang (2016). Introduction to Linear Algebra. *Wellesley-Cambridge Press*.
- Axler (2024). Linear Algebra Done Right. *Springer*. <https://doi.org/10.1007/978-3-031-41026-0>
- Luxburg (2007). A tutorial on spectral clustering. *Statist. Comput.*. <https://doi.org/10.1007/s11222-007-9033-z>

## Cite this entry

```bibtex
@misc{dictml_matrix,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {matrix},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/matrix.html}
}
```


---

# mean

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/mean.html) · [typeset PDF](https://dictionaryofml.org/terms/mean.pdf) · [LaTeX](https://dictionaryofml.org/terms/mean.tex) · [Python demo](https://dictionaryofml.org/terms/mean.py) · [notebook](https://dictionaryofml.org/terms/mean.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

The mean of a random vector ${\mathbf x}$ is its expectation, the Lebesgue integral of ${\mathbf x}$ with respect to its probability distribution. The term also refers to the sample mean, i.e., the average of the data points in a dataset. The two usages are consistent: the sample mean is the mean of the random vector obtained by drawing a data point uniformly from the dataset. Both usages share a further characterization: for a random vector with a finite second moment, the mean is the unique solution of a risk minimization problem under the squared error loss. For the random vector from a dataset, this optimization problem reduces to empirical risk minimization in the simplest regression setting: predicting a numeric label without features, for which the learned hypothesis is the sample mean of the labels.

## Definition

A weather station records the daily maximum temperature for a year. Adding the $365$ numbers and dividing by $365$ returns a single value that summarizes the whole record. This is the mean of the recorded numbers, and the same word names the corresponding quantity for a probability distribution.

The mean of a random vector ${\mathbf x}$, which takes on values in a Euclidean space $\mathbb{R}^{d}$, is its expectation $\mathbb{E}\{{\mathbf x}\}$. It is the Lebesgue integral of ${\mathbf x}$ with respect to its probability distribution $\mathbb{P}$ (e.g., see Billingsley (1986, Thm. 16.12, Eq. (21.17)) or Rudin (1987)), i.e.,

$$
\mathbb{E}\{{\mathbf x}\} = \int_{\mathbb{R}^{d}} {\mathbf x} \, \mathrm{d}\mathbb{P}({\mathbf x}) \text{.}
$$

Here, $\mathbb{P}$ is the probability distribution that ${\mathbf x}$ induces on $\mathbb{R}^{d}$, so the integral runs over the values ${\mathbf x}$ takes rather than over the underlying sample space. No probability density function need exist for the mean to be defined; when one does, denoted by $p$, it is $\int_{\mathbb{R}^{d}} {\mathbf x} \, p({\mathbf x}) \, \mathrm{d}{\mathbf x}$ (Billingsley, 1986, Thm. 16.10).

The average of the opening paragraph is the sample mean of a finite dataset of sample size $m$, $\mathcal{D} = \left\{ {\mathbf x}^{(1)}, \,\ldots, \,{\mathbf x}^{(m)} \in \mathbb{R}^{d}\right\}$. These two usages are consistent. A dataset defines a discrete random variable $\widetilde{{\mathbf x}}^{(\mathcal{D})}={\mathbf x}^{(I)}$ on the sample space $\{1, \,\ldots, \,m\}$. Here, the index $I$ is chosen uniformly at random, i.e., $\mathbb{P}\left(I=r\right)=1/m$ for all $r=1,\ldots,m$. The mean of $\widetilde{{\mathbf x}}^{(\mathcal{D})}$ is precisely the sample mean

$$
({1}/{m}) \sum_{r=1}^{m}
{\mathbf x}^{(r)} \text{.}
$$

The probability distribution of $\widetilde{{\mathbf x}}^{(\mathcal{D})}$, which places mass $1/m$ on each data point, is the empirical distribution of the dataset. Fig. 1 shows both readings for a real-valued random variable and a dataset of seven numbers, with the empirical distribution drawn as its cumulative distribution function.

![Figure 1 of the entry mean](https://dictionaryofml.org/terms/fig/mean-fig1.svg)

*Figure 1: The two readings of *mean*. (a) A real-valued random variable with the standard normal probability density function has mean $\mathbb{E}\{x\} = 0$, the Lebesgue integral of $x$ against that probability density function. (b) The $m = 7$ data points average to the sample mean $0.1$. (c) The empirical distribution of the same data points, drawn as its cumulative distribution function $F^{(\mathcal{D})}\left(x\right)$: a staircase rising by $1/m$ at each data point, from $0$ to $1$. It is the probability distribution of the discrete random variable $\widetilde{{\mathbf x}}^{(\mathcal{D})}$ above, and its mean is the sample mean of (b). The sample mean $0.1$ of (b) and (c) differs from the mean $0$ of the random variable in (a) because $m$ is finite*

The two readings are further connected by a shared characterization. Consider any random vector with a finite second moment, i.e., $\mathbb{E}\{ \left\lVert {\mathbf x} \right\rVert_{2}^{2} \} < \infty$. Its mean is the unique solution of the following risk minimization problem (Bertsekas and Tsitsiklis, 2008):

$$
\mathbb{E}\{{\mathbf x}\} = \mathop{\mathrm{arg\,min}}_{{\mathbf c} \in \mathbb{R}^{d}}
\mathbb{E} \big\{\left\lVert {\mathbf x} - {\mathbf c} \right\rVert_{2}^{2}\big \} \text{.}
$$

The objective function of this problem is a strictly convex function of the optimization variable ${\mathbf c} \in \mathbb{R}^{d}$.

For the discrete random variable that a dataset defines, the risk is the average squared error over the dataset, so this risk-minimization characterization applies directly to the simplest machine learning problem, predicting a numeric label without any features. Consider a training set $\mathcal{D}^{(\rm train)} = \big\{ y^{(1)}, \,\ldots, \,y^{(m)} \big\}$ that consists of labels $y^{(r)} \in \mathbb{R}$ only. A hypothesis is then a single number $h \in \mathbb{R}$ that serves as the prediction for every data point. Empirical risk minimization with the squared error loss searches over candidate hypotheses $h' \in \mathbb{R}$ and learns

$$
\widehat{h}
= \mathop{\mathrm{arg\,min}}_{h' \in \mathbb{R}}
\frac{1}{m} \sum_{r=1}^{m}
\big(y^{(r)} - h'\big)^{2}
= \frac{1}{m} \sum_{r=1}^{m}
y^{(r)} \text{,}
$$

i.e., the learned hypothesis is the sample mean of the labels (see Fig. 2). This setting coincides with linear regression for data points that carry the constant scalar feature $x = 1$.

![Figure 2 of the entry mean](https://dictionaryofml.org/terms/fig/mean-fig2.svg)

*Figure 2: A training set of $m = 5$ numeric labels $y^{(r)}$ (filled circles) and the learned hypothesis $\widehat{h} = 4$ (horizontal dashed line), which is the sample mean of the labels. The vertical gray segments are the errors $y^{(r)} - \widehat{h}$; the sample mean minimizes the average of their squares*

Squaring the deviations also makes the mean sensitive to outliers. Moving one of $m$ data points by an amount $\delta$ shifts the sample mean by $\delta / m$, so a single corrupted data point can move the mean arbitrarily far. The trimmed mean that discards a fraction $\gamma$ at each end stays bounded however far at most $\gamma m$ data points are moved, since the moved data points end up among the discarded ones. The median stays bounded however far fewer than half of the $m$ data points are moved, since the unmoved data points are then a majority and the two middle values of the sorted list lie between the smallest and the largest of them. Both are therefore preferred when features or labels may be unreliable (Huber, 1992; pythondemos/mean.py).

**Synonyms:** expectation, expected value.

**See also:** [random variable](https://dictionaryofml.org/terms/rv.html), random vector, expectation, sample mean, [probability distribution](https://dictionaryofml.org/terms/probdist.html), Lebesgue integral, [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), squared error loss, [linear regression](https://dictionaryofml.org/terms/linreg.html), median, trimmed mean.

## References

- Billingsley (1986). Probability and Measure. *Wiley*.
- Bertsekas and Tsitsiklis (2008). Introduction to Probability. *Athena Scientific*.
- Huber (1992). Robust estimation of a location parameter. *Breakthroughs in Statistics: Methodology and Distribution*.
- Rudin (1987). Real and Complex Analysis. *McGraw-Hill*.

## Cite this entry

```bibtex
@misc{dictml_mean,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {mean},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/mean.html}
}
```


---

# membership inference attack

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-09. [HTML](https://dictionaryofml.org/terms/membershipinferenceattack.html) · [typeset PDF](https://dictionaryofml.org/terms/membershipinferenceattack.pdf) · [LaTeX](https://dictionaryofml.org/terms/membershipinferenceattack.tex) · [Python demo](https://dictionaryofml.org/terms/membershipinferenceattack.py) · [notebook](https://dictionaryofml.org/terms/membershipinferenceattack.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A membership inference attack aims at determining whether a particular data point was used to train a machine learning model. The basic idea is that a learned hypothesis delivers more accurate predictions for data points that are similar to those used for training. An adversary queries the learned hypothesis and compares the loss of its prediction against a threshold. Mitigation strategies include regularization, which makes a machine learning method generalize beyond the training set: the learned hypothesis then behaves similarly for data points in the training set and for those outside it, which makes it harder for an adversary to guess the data points of the training set.

## Definition

A volunteer observer at Krems an der Donau reports the minimum and the maximum temperature for selected days. Each day is one data point ${\mathbf z} = \left( x,y \right)$, whose feature $x$ is that day's minimum temperature and whose label $y$ is that day's maximum temperature, and the reported days form the training set $\mathcal{D}^{(\rm train)}$. A company learns a hypothesis $\widehat{h}$ from $\mathcal{D}^{(\rm train)}$ to predict the maximum temperature of a day from its minimum. The company publishes $\widehat{h}$, so anyone can query it at any minimum temperature. The GeoSphere weather station at Krems publishes the minimum and the maximum temperature of every day. Anyone with both can try to determine whether the report of a specific day was used for training. Since the reports come from one person, a yes tells that the observer was around Krems an der Donau on that day. Whether the observer was at Krems on a specific day is personal data, even though a temperature is not (European Parliament and Council of the European Union, 2016, Art. 4). This scenario is an example of a membership inference attack: a privacy attack where an adversary tries to determine whether a particular data point was used to train a machine learning model (Shokri et al., 2017; Vassilev et al., 2024, Sect. 2.4.2).

For a specific day, the adversary reads that day's $x$ and $y$ from the station's record, computes the prediction $\widehat{h}(x)$, and compares it against $y$. A loss $\big(y - \widehat{h}(x)\big)^{2}$ below a threshold indicates that the report of that day was used for training.

Fig. 1 carries this attack out on the 31 days of October 2024 at Krems. One threshold labels all 31 days correctly: the loss is zero on every day the observer reported and never below $0.23\,^{\circ}$C$^{2}$ on any other day. The days it declares reported form two blocks of consecutive days, 7 to 13 and 20 to 26 October, which is when the observer was there. The hypothesis behind the figure is obtained by linear regression on the features $x'_{j} = \cos\big((j-1) \pi u\big)$ for $j = 1, \ldots, d$, with $u$ the minimum temperature $x$ rescaled to $[0,1]$ over the month; for $d = 14$, one feature per reported day, least squares reproduces every reported day exactly.

![Figure 1 of the entry membershipinferenceattack](https://dictionaryofml.org/terms/fig/membershipinferenceattack-fig1.svg)

*Figure 1: The loss the published hypothesis incurs on each day of October 2024 at Krems an der Donau, computed by an adversary from the weather station's own record. Every day that the observer did not report carries a solid stem, drawn from zero to its loss and topped by a disc. The 14 days the observer did report carry no stem at all: the squares resting on the horizontal axis mark a loss of zero. The days those squares mark are 7 to 13 and 20 to 26 October, the two stays, so the losses say not only which days were used for training but also when the observer was at Krems. Data generated by pythondemos/membershipinferenceattack.py*

The number $d$ sets the effective dimension of the underlying hypothesis space and so how far the machine learning method can overfit. Fig. 2 plots how much the attack gains from that. Fix a threshold $\tau$ on the loss. Of the 14 days the observer reported, the fraction incurring a loss of at most $\tau$ is the detection rate. Of the 17 days of the month the observer did not report, the fraction incurring a loss of at most $\tau$ is the false-alarm rate. The advantage of the attack is the largest value of the detection rate minus the false-alarm rate over all thresholds. An adversary that answers yes on every day, or no on every day, leaves the two rates equal and the advantage zero, which is why the advantage measures what the attack gains over guessing, and a threshold that separates the two groups makes it one. The threshold is chosen with the answer already known, so the advantage is the most any threshold could achieve rather than what an adversary would obtain. With two features, it is $0.10$, and it reaches one once $d$ reaches the number of reported days.

![Figure 2 of the entry membershipinferenceattack](https://dictionaryofml.org/terms/fig/membershipinferenceattack-fig2.svg)

*Figure 2: How the attack of Fig. 1 improves as the hypothesis space grows. The curve is the advantage, the largest difference over all thresholds between the fraction of the 14 reported days whose loss falls below the threshold and the fraction of the 17 other days whose loss does, for a hypothesis learned from the same reported days with $d$ features. Data generated by pythondemos/membershipinferenceattack.py*

The attack succeeds only to the extent that the learned hypothesis incurs a smaller loss inside the training set than outside it, which is what the generalization gap measures. Under a probabilistic model for how the data points and their membership of the training set arise, the success of such an attack is tied to that gap. The quantity is the same difference as above, with the two rates replaced by probabilities under that model. For a loss bounded by a constant $B$, an adversary that answers yes with a probability decreasing in the loss has a difference that grows in proportion to the expected generalization gap (Yeom et al., 2018, Thm. 2). Regularization controls that gap: for a convex loss that is Lipschitz with constant $L$, adding the penalty term $\alpha \left\lVert {\mathbf w} \right\rVert_{2}^{2}$ bounds the expected generalization gap by $2L^{2}/(\alpha m)$ (Shalev-Shwartz and Ben-David, 2014, Cor. 13.6). Fig. 3 repeats the attack against the same features learned with that penalty term, and the picture no longer tells the two groups apart. Regularization therefore serves two purposes, accurate predictions outside the training set (Hastie et al., 2009, Sect. 7.4) and a defence against a membership inference attack (Shokri et al., 2017, Sect. VIII). For the observer at Krems, that is the difference between a published hypothesis that reveals nothing about them and one that reveals the days they were in town.

![Figure 3 of the entry membershipinferenceattack](https://dictionaryofml.org/terms/fig/membershipinferenceattack-fig3.svg)

*Figure 3: The same attack as in Fig. 1, on the same axes, against the hypothesis learned from the same 14 reported days with the penalty term $\alpha \left\lVert {\mathbf w} \right\rVert_{2}^{2}$ for $\alpha = 10$. The reported days now carry dashed stems topped by a square, of much the same height as the solid stems topped by a disc that mark the days the observer did not report, so no threshold picks them out. Their average loss rises from $0$ to $5.24\,^{\circ}$C$^{2}$ while the average loss elsewhere falls from $14.29$ to $5.26\,^{\circ}$C$^{2}$: the generalization gap closes to $0.02\,^{\circ}$C$^{2}$ and the adversary's advantage drops from one to at most $0.15$. Data generated by pythondemos/membershipinferenceattack.py*

**See also:** attack, privacy attack, personal data, [regularization](https://dictionaryofml.org/terms/regularization.html), differential privacy, [overfitting](https://dictionaryofml.org/terms/overfitting.html), [training error](https://dictionaryofml.org/terms/trainerr.html), [validation error](https://dictionaryofml.org/terms/valerr.html), [training set](https://dictionaryofml.org/terms/trainset.html).

## References

- European Parliament and Council of the European Union (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (Text with EEA relevance). <https://eur-lex.europa.eu/eli/reg/2016/679/oj>
- Shokri et al. (2017). Membership Inference Attacks Against Machine Learning Models. *2017 IEEE Symp. Secur. Privacy*. <https://doi.org/10.1109/SP.2017.41>
- Vassilev et al. (2024). Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations. <https://doi.org/10.6028/NIST.AI.100-2e2023>
- Yeom et al. (2018). Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting. *2018 IEEE 31st Computer Security Foundations Symposium (CSF)*. <https://doi.org/10.1109/CSF.2018.00027>
- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>

## Cite this entry

```bibtex
@misc{dictml_membershipinferenceattack,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {membership inference attack},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/membershipinferenceattack.html}
}
```


---

# machine learning pipeline (ML pipeline)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/mlpipeline.html) · [typeset PDF](https://dictionaryofml.org/terms/mlpipeline.pdf) · [LaTeX](https://dictionaryofml.org/terms/mlpipeline.tex) · [Python demo](https://dictionaryofml.org/terms/mlpipeline.py) · [notebook](https://dictionaryofml.org/terms/mlpipeline.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A machine learning pipeline (ML pipeline) is the composition of the functions by which a machine learning system turns raw data into predictions. Its stages typically are data preprocessing, feature learning or hand-designed feature extraction, model training, and inference on new data points. Each stage is a function whose output is the input of the next, so the whole pipeline is one map from raw data to predictions and is applied identically during training and in deployment. Fitting stages such as feature scaling inside the pipeline, on the training set only, prevents data leakage from the validation set or test set.

## Definition

The weather station Krems an der Donau reports eight measurements of each day to the GeoSphere Austria data hub. Turning that stream into a forecast of tomorrow's minimum temperature takes four steps: cleaning the records, extracting features, training a model, and computing the prediction. The sequence of such steps is an ML pipeline: a composition (i.e., concatenation) of several functions within a machine learning system. The individual functions include data preprocessing, feature learning or hand-designed feature extraction, model training, and inference on new data points. By combining them, a machine learning system turns raw data into predictions (Hapke and Nelson, 2020) (Fig. 1).

Written as math, a pipeline of three stages is the concatenation

\begin{equation}
\tag{1}
h = h^{(3)} \circ h^{(2)}
\circ h^{(1)}
\text{,}
\end{equation}

with $h^{(1)}$ the feature map, $h^{(2)}$ the hypothesis applied to the resulting features, and $h^{(3)}$ the post-processing of its output. Of the four steps above, cleaning and feature extraction form $h^{(1)}$, the trained hypothesis is $h^{(2)}$ (training is how it is chosen, not a factor of $h$), and the post-processing of the prediction is $h^{(3)}$. Each factor is chosen from a hypothesis space of its own, $h^{(j)} \in \mathcal{H}^{(j)}$ for $j = 1, 2, 3$. The concatenations Eq. (1) are themselves a hypothesis space,

$$
\mathcal{H} = \bigl\{
h^{(3)} \circ h^{(2)} \circ
h^{(1)} \; : \;
h^{(j)} \in \mathcal{H}^{(j)},
\; j = 1, 2, 3
\bigr\}
\text{,}
$$

so training the pipeline is empirical risk minimization over $\mathcal{H}$ rather than over any one stage.

Each stage is a function whose output is the input of the next, so the pipeline is one map from raw data to predictions, applied identically during training and in deployment. Stages with their own parameters, such as feature scaling, are fitted on the training set only. Fitting them on all available data leaks information from the validation set or the test set into the learned hypothesis, so that the validation error or the error on the test set underestimates the risk (see data leakage) (Hastie et al., 2009, Sect. 7.10.2). Software libraries therefore bundle the stages into one object that is trained as a whole on the training set (Pedregosa et al., 2011). The pipeline is the computational part of a machine learning system, which additionally comprises data storage, deployment, and monitoring.

![Figure 1 of the entry mlpipeline](https://dictionaryofml.org/terms/fig/mlpipeline-fig1.svg)

*Figure 1: An ML pipeline as a concatenation of functions Eq. (1), the output of each stage the input of the next, with the Krems forecast of the text named under each stage. The three stages drawn with a dashed border carry parameters of their own and are fitted on the training set only, which is what prevents data leakage from the validation set or test set into the learned hypothesis*

**See also:** function, machine learning system, preprocessing, [feature learning](https://dictionaryofml.org/terms/featlearn.html), inference, [data leakage](https://dictionaryofml.org/terms/dataleakage.html), [training set](https://dictionaryofml.org/terms/trainset.html), [hypothesis space](https://dictionaryofml.org/terms/hypospace.html), [empirical risk minimization](https://dictionaryofml.org/terms/erm.html).

## References

- Hapke and Nelson (2020). Building Machine Learning Pipelines: Automating Model Life Cycles with TensorFlow. *O'Reilly Media*.
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Pedregosa et al. (2011). Scikit-learn: Machine Learning in Python. *Journal of Machine Learning Research*. <https://www.jmlr.org/papers/v12/pedregosa11a.html>

## Cite this entry

```bibtex
@misc{dictml_mlpipeline,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {machine learning pipeline (ML pipeline)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/mlpipeline.html}
}
```


---

# model

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-10-29. [HTML](https://dictionaryofml.org/terms/model.html) · [typeset PDF](https://dictionaryofml.org/terms/model.pdf) · [LaTeX](https://dictionaryofml.org/terms/model.tex) · [Python demo](https://dictionaryofml.org/terms/model.py) · [notebook](https://dictionaryofml.org/terms/model.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

In machine learning, the word *model* is used in different ways. Formally, a model is a hypothesis space $\mathcal{H}$: the set of candidate hypothesis maps from which a machine learning method picks (or learns) one. In applied machine learning literature and in software libraries, the word *model* also refers to the trained predictor, i.e., to the learned hypothesis returned by a machine learning algorithm; the EU AI Act uses this sense in the compound term general-purpose AI model. A different type of mathematical model is a probabilistic model, which specifies a family of probability distributions that characterize a data generation process.

## Definition

Consider the problem of predicting tomorrow's maximum daytime temperature from today's morning temperature. Machine learning methods learn a hypothesis map $\widehat{h}$ that maps today's morning temperature, the feature, to an accurate prediction of tomorrow's maximum temperature, the label. This learned hypothesis is chosen from a (typically large) set of candidate hypothesis maps, i.e., a hypothesis space $\mathcal{H}$.

The hypothesis space underlying a machine learning method is its model: the set of candidate hypothesis maps, chosen in advance, from which the method picks one (Shalev-Shwartz and Ben-David, 2014, Sect. 2.3). For example, linear regression and logistic regression use the linear model as their hypothesis space. A decision tree method uses a hypothesis space that consists of hypothesis maps generated by a flow chart. Large language model systems use a hypothesis space consisting of non-linear maps represented by an artificial neural network.

Model training refers to the process of finding a hypothesis in the hypothesis space that yields accurate predictions. A trained (or fitted) model is a learned hypothesis $\widehat{h} = \mathcal{A}(\mathcal{D}^{(\rm train)})$, the output of the map $\mathcal{A}$ by which the machine learning method turns a training set $\mathcal{D}^{(\rm train)}$ into a hypothesis. In applied machine learning literature and in software libraries, the word *model* typically refers to a trained model, i.e., to a specific hypothesis map $\widehat{h}$ (Goodfellow et al., 2016, Sect. 5.1; Murphy, 2012, Sect. 3.5.2; Prince, 2023). The EU AI Act uses the word *model* in the same sense (i.e., a learned hypothesis) in the compound term general-purpose AI model (European Parliament and Council of the European Union, 2024, Art. 3(63)).

A different type of mathematical model is a probabilistic model, which specifies a family of probability distributions. Each such probability distribution defines a data generator: data points are realizations of independent and identically distributed RVs with the given probability distribution.

Fig. 1 illustrates these three uses of the word *model*: the hypothesis space, the trained model, and the probabilistic model.

![Figure 1 of the entry model](https://dictionaryofml.org/terms/fig/model-fig1.svg)

*Figure 1: Three uses of the word *model* in machine learning practice. (a) Hypothesis space consisting of three linear maps (solid, dashed, dotted). (b) Trained model: the learned hypothesis $\widehat{h} = \mathcal{A}(\mathcal{D}^{(\rm train)})$ selected from the hypothesis space in (a) by a machine learning map $\mathcal{A}$ acting on a training set $\mathcal{D}^{(\rm train)}$ of data points (blue circles). (c) Probabilistic model consisting of two probability distributions (solid and dashed outlines) for the generation of data points with feature $x$ and label $y$*

**Synonyms:** hypothesis space, hypothesis class, model class.

**See also:** [hypothesis space](https://dictionaryofml.org/terms/hypospace.html), [hypothesis](https://dictionaryofml.org/terms/hypothesis.html), [training](https://dictionaryofml.org/terms/training.html), probabilistic model, parametric model, model parameter, general-purpose AI model.

## References

- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Goodfellow et al. (2016). Deep Learning. *MIT Press*.
- Murphy (2012). Machine Learning: A Probabilistic Perspective. *MIT Press*.
- Prince (2023). Understanding Deep Learning. *MIT Press*.
- European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act) (Text with EEA relevance). <https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng>

## Cite this entry

```bibtex
@incollection{dictml_model,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {model},
  booktitle = {Dictionary of Applied Machine Learning},
  publisher = {Springer Nature Singapore},
  year = {2026},
  doi = {10.1007/978-981-92-2012-0_2-1},
  isbn = {978-981-92-2012-0},
  url = {https://doi.org/10.1007/978-981-92-2012-0_2-1}
}
```


---

# model selection

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-10. [HTML](https://dictionaryofml.org/terms/modelsel.html) · [typeset PDF](https://dictionaryofml.org/terms/modelsel.pdf) · [LaTeX](https://dictionaryofml.org/terms/modelsel.tex) · [Python demo](https://dictionaryofml.org/terms/modelsel.py) · [notebook](https://dictionaryofml.org/terms/modelsel.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Model selection chooses among several alternative machine learning methods by comparing their validation errors. Each machine learning method learns a hypothesis via a training method such as empirical risk minimization. A core design choice of a machine learning method is the underlying hypothesis space, or model. The effective hypothesis space can also depend on hyperparameters of the optimization algorithms used to solve empirical risk minimization. The training error of a machine learning method cannot serve as the criterion for model selection, as it always chooses the largest of nested hypothesis spaces. The validation error can be obtained from the average loss on a single validation set, or via k-fold cross-validation. The ultimately chosen hypothesis must be assessed on data points that have been used neither for learning any individual hypothesis nor for choosing among them.

## Definition

Fig. 1 depicts data points collected by a weather station at Krems an der Donau. Each data point represents a day, with the morning minimum temperature as feature $x$ and the maximum daytime temperature as label $y$. The figure also shows two hypothesis maps, each learned by a machine learning method via empirical risk minimization from the training set $\mathcal{D}^{(\rm train)}$ of six data points. One is a linear hypothesis $\widehat{h}^{(1)}$, learned over a linear model. The other is a degree-five polynomial $\widehat{h}^{(2)}$, learned over the larger hypothesis space of polynomial regression.

The polynomial delivers perfect predictions on every data point of $\mathcal{D}^{(\rm train)}$, so its training error, the average squared error loss over $\mathcal{D}^{(\rm train)}$, is zero. The linear hypothesis misses each of them and its training error is $0.47$. The training error therefore ranks the polynomial first, and following that ranking would pick the candidate that predicts worst on a data point outside $\mathcal{D}^{(\rm train)}$. Model selection ranks the two instead by the validation error on a validation set $\mathcal{D}^{(\rm val)}$ that empirical risk minimization did not use. On the fourteen validation data points of Fig. 1, the average squared error loss is $2065$ for the polynomial and $7.06$ for the linear hypothesis, so model selection picks the linear hypothesis $\widehat{h}^{(1)}$.

The validation error of the selected hypothesis is biased downward, since the selection favored the candidate with the smallest one. Model selection therefore leaves a test set $\mathcal{D}^{(\rm test)}$ untouched for estimating the risk of the selected hypothesis.

![Figure 1 of the entry modelsel](https://dictionaryofml.org/terms/fig/modelsel-fig1.svg)

*Figure 1: Model selection between two machine learning methods that learn a hypothesis via empirical risk minimization from the same six data points (filled circles). One method uses degree-five polynomials as hypothesis space, the other a linear model. The degree-five polynomial $\widehat{h}^{(2)}$ (dashed curve) delivers perfect predictions on every data point of the training set, so its training error is zero against $0.47$ for the linear hypothesis $\widehat{h}^{(1)}$ (solid curve). Averaged over the fourteen validation data points (open triangles), the squared error loss is $2065$ for the polynomial and $7.06$ for the linear hypothesis, so the validation error selects the linear hypothesis $\widehat{h}^{(1)}$. The six data points of the test set (open squares) are held out from both empirical risk minimization and the model selection, so the average squared error loss on them, $1.83$, estimates the risk of the selected hypothesis. Data generated by pythondemos/validation.py*

Model selection chooses among several alternative machine learning methods by comparing their validation errors (Shalev-Shwartz and Ben-David, 2014, Sect. 11.2.2). Consider $C$ candidates $\widehat{h}^{(c)}$, $c = 1, \ldots, C$, obtained from empirical risk minimization with a different effective hypothesis space, or model, $\mathcal{H}^{(c)}$. The effective hypothesis space can also depend on hyperparameters of the optimization algorithm used to solve empirical risk minimization. Typical hyperparameters include the learning rate of a gradient-based method, the strength $\alpha$ of a regularization penalty term, and the depth of a decision tree (Goodfellow et al., 2016, Sect. 11.4).

The training error of the learned hypotheses $\widehat{h}^{(c)}$ cannot serve as the criterion for choosing between them. For nested candidates $\mathcal{H}^{(1)} \subseteq \cdots \subseteq \mathcal{H}^{(C)}$ it always chooses the largest hypothesis space, since the minimum of the training error over a superset is no larger, even when that results in overfitting, as the degree-five polynomial of Fig. 1 does (Hastie et al., 2009, Sect. 7.2, Fig. 7.1; Shalev-Shwartz and Ben-David, 2014, Sect. 11.2.3). Model selection therefore follows model training, e.g., via empirical risk minimization, and model validation, e.g., via k-fold cross-validation.

Algorithm 1 summarizes a complete machine learning workflow, consisting of model training, model validation and model selection.

![Algorithm 1 of the entry modelsel](https://dictionaryofml.org/terms/fig/modelsel-alg1.svg)

*Algorithm 1: Model selection*

The two loss functions used in Algorithm 1 for training and validation can be different: empirical risk minimization needs one that its optimization method can handle, while validation can use the loss function that matters for the application, such as the 0/1 loss. When the training set is too small to spare a validation set, steps 2 and 3 of Algorithm 1 can be replaced by k-fold cross-validation (Hastie et al., 2009, Sect. 7.10).

The final hypothesis $\widehat{h}^{(c^{\star})}$ delivered by Algorithm 1 must be evaluated on a test set. A test set consists of data points that have been used neither for learning any individual $\widehat{h}^{(c)}$ nor for choosing between them. Under an independent and identically distributed assumption for the data generation, the average loss on $\mathcal{D}^{(\rm test)}$ is an unbiased estimator of the risk of $\widehat{h}^{(c^{\star})}$: the data points of $\mathcal{D}^{(\rm test)}$ are independent of how $\widehat{h}^{(c^{\star})}$ was obtained (Shalev-Shwartz and Ben-David, 2014, Sect. 11.2.1), so each loss has the risk as its expectation and so has their average (Bertsekas and Tsitsiklis, 2008, Sect. 9.1). If $\mathcal{D}^{(\rm train)}$ and $\mathcal{D}^{(\rm val)}$ are modelled as random, so is $\widehat{h}^{(c^{\star})}$, and with it the risk being estimated: the expectation is then conditional on $\widehat{h}^{(c^{\star})}$, over $\mathcal{D}^{(\rm test)}$ alone.

**See also:** validation, [validation error](https://dictionaryofml.org/terms/valerr.html), [$k$-fold cross-validation](https://dictionaryofml.org/terms/kfoldcv.html), [training set](https://dictionaryofml.org/terms/trainset.html), [validation set](https://dictionaryofml.org/terms/valset.html), [test set](https://dictionaryofml.org/terms/testset.html), [hyperparameter](https://dictionaryofml.org/terms/hyperparameter.html), [hypothesis space](https://dictionaryofml.org/terms/hypospace.html), [overfitting](https://dictionaryofml.org/terms/overfitting.html).

## References

- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Goodfellow et al. (2016). Deep Learning. *MIT Press*.
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Bertsekas and Tsitsiklis (2008). Introduction to Probability. *Athena Scientific*.

## Cite this entry

```bibtex
@misc{dictml_modelsel,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {model selection},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/modelsel.html}
}
```


---

# norm

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/norm.html) · [typeset PDF](https://dictionaryofml.org/terms/norm.pdf) · [LaTeX](https://dictionaryofml.org/terms/norm.tex) · [Python demo](https://dictionaryofml.org/terms/norm.py) · [notebook](https://dictionaryofml.org/terms/norm.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A norm on a vector space is a function that assigns each vector a nonnegative number, its length. A norm is definite and homogeneous, and it satisfies the triangle inequality. Every norm defines a metric via the norm of the difference of two vectors, and every inner product induces a norm, but not every norm is induced by one. The $\ell_{p}$-norms on $\mathbb{R}^{d}$ include the $\ell_{1}$-norm, the Euclidean norm, and the $\ell_{\infty}$-norm; among them only the Euclidean norm is induced by an inner product. In machine learning, norms are used to construct loss functions and regularizers.

## Definition

The hourly temperature recordings of one day at a weather station form the feature vector ${\mathbf x} \in \mathbb{R}^{d}$ of that day, with one entry per hour. Many machine learning methods use a measure for the size of such a vector or for the distance between two of them. A norm provides both: it assigns each vector a length, and the length of the difference of two vectors is their distance.

Formally, a norm $\left\lVert \, \cdot \, \right\rVert_{}$ on a vector space $\mathcal{V}$ over the field $\mathbb{R}$ is a function $\left\lVert \, \cdot \, \right\rVert_{}: \mathcal{V} \to \mathbb{R}_{+}$ that satisfies the conditions below (Horn and Johnson, 2013). These conditions hold for all vectors ${\mathbf u}, {\mathbf v} \in \mathcal{V}$ and all scalars $\beta \in \mathbb{R}$, with $|\beta|$ the absolute value of $\beta$:

- Definiteness: $\left\lVert {\mathbf u} \right\rVert_{} = 0 \iff {\mathbf u} = \mathbf{0}$;
- Homogeneity: $\left\lVert \beta {\mathbf u} \right\rVert_{} = |\beta| \cdot \left\lVert {\mathbf u} \right\rVert_{}$;
- Triangle inequality: $\left\lVert {\mathbf u} + {\mathbf v} \right\rVert_{} \leq \left\lVert {\mathbf u} \right\rVert_{} + \left\lVert {\mathbf v} \right\rVert_{}$.

The same conditions define a norm on a vector space over the field of complex numbers $\mathbb{C}$, with $\beta \in
\mathbb{C}$ and $|\beta|$ its modulus. A function that satisfies homogeneity and the triangle inequality but assigns the value zero to some nonzero vector is a seminorm, not a norm: on $\mathbb{R}^{2}$, the map ${\mathbf u} = (u_{1}, u_{2})^{\top} \mapsto |u_{1}|$, which keeps only the first entry, is a seminorm that is zero on $(0,1)^{\top}$.

Every norm defines a metric by $d({\mathbf u},{\mathbf v}) := \left\lVert {\mathbf u} - {\mathbf v} \right\rVert_{}$, making $\left( \mathcal{V},d(\cdot,\cdot) \right)$ a metric space. Some norms are induced by an inner product $\langle \cdot, \cdot \rangle$, a function of two vectors that is symmetric, linear in each argument, and positive-definite (on $\mathbb{R}^{d}$, $\langle {\mathbf u}, {\mathbf v} \rangle = {\mathbf u}^{\top} {\mathbf v}$). On an inner product space, $\left\lVert {\mathbf u} \right\rVert_{} := \sqrt{\langle {\mathbf u}, {\mathbf u} \rangle}$ is a norm (Axler, 2024, Thm. 6.14 and Thm. 6.17), and the norm of a Hilbert space is of this form. An induced norm satisfies the parallelogram identity $\left\lVert {\mathbf u} + {\mathbf v} \right\rVert_{}^{2} + \left\lVert {\mathbf u} - {\mathbf v} \right\rVert_{}^{2}
= 2 \left\lVert {\mathbf u} \right\rVert_{}^{2} + 2 \left\lVert {\mathbf v} \right\rVert_{}^{2}$, obtained by expanding $\langle {\mathbf u} \pm {\mathbf v}, {\mathbf u} \pm {\mathbf v} \rangle$.

For the vector space $\mathcal{V} = \mathbb{R}^{d}$ of feature vectors ${\mathbf x} = (x_{1}, \ldots,
x_{d})^{\top}$ and a number $p \geq 1$, the $\ell_p$-norm of ${\mathbf x}$ is defined as $\left\lVert {\mathbf x} \right\rVert_{p} = \left( \displaystyle \sum_{j=1}^{d} |x_{j}|^p \right)^{1/p}$. It satisfies the triangle inequality by Minkowski's inequality and is therefore a norm (Horn and Johnson, 2013, Sect. 5.2, Eq. (5.2.4)). Its instances include the $\ell_1$-norm $\left\lVert {\mathbf x} \right\rVert_{1} = \sum_{j=1}^{d} |x_{j}|$ and the $\ell_2$-norm $\left\lVert {\mathbf x} \right\rVert_{2} = \big( \sum_{j=1}^{d} x_{j}^{2} \big)^{1/2}$ (the Euclidean norm); the limit $p \to \infty$ yields the $\ell_\infty$-norm $\left\lVert {\mathbf x} \right\rVert_{\infty} = \max_{j = 1, \ldots, d} |x_{j}|$. The $\ell_2$-norm is induced by the inner product ${\mathbf u}^{\top} {\mathbf v}$, and it is the only $\ell_p$-norm induced by an inner product. For ${\mathbf u} = (1, 0)^{\top}$ and ${\mathbf v} = (0, 1)^{\top}$ in $\mathbb{R}^{2}$, both ${\mathbf u} + {\mathbf v}$ and ${\mathbf u} - {\mathbf v}$ have $\ell_p$-norm $2^{1/p}$, so the left-hand side of the parallelogram identity is $2 \cdot 2^{2/p}$ while the right-hand side is $4$, and the two agree only for $p = 2$. The left-hand side equals $8$ for the $\ell_1$-norm and $2$ for the $\ell_\infty$-norm. The three norms differ in which vectors they count as having unit length. The unit sphere $\mathbb{S}^{(1)} := \big\{ {\mathbf u} \in \mathbb{R}^{2} : \left\lVert {\mathbf u} \right\rVert_{} = 1 \big\}$ of a norm on $\mathbb{R}^{2}$ contains those vectors, and Fig. 1 shows it for the $\ell_1$-, the $\ell_2$-, and the $\ell_\infty$-norm.

![Figure 1 of the entry norm](https://dictionaryofml.org/terms/fig/norm-fig1.svg)

*Figure 1: Unit spheres $\mathbb{S}^{(1)}$ of the $\ell_p$-norms on $\mathbb{R}^{2}$, for vectors ${\mathbf u} = (u_{1}, u_{2})^{\top}$ and $p=1$ (dashed diamond), $p=2$ (solid circle), and $p=\infty$ (dotted square)*

Machine learning methods use norms to construct loss functions and regularizers. In linear regression, the label vector ${\mathbf y} \in \mathbb{R}^{m}$ stacks the labels of a training set of size $m$ and the feature matrix ${\mathbf X} \in \mathbb{R}^{m \times d}$ stacks its feature vectors. The average squared error loss of a linear hypothesis with model parameters ${\mathbf w}$ is then the scaled squared Euclidean norm $(1/m) \left\lVert {\mathbf y} - {\mathbf X} {\mathbf w} \right\rVert_{2}^{2}$ of the residual ${\mathbf y} - {\mathbf X} {\mathbf w}$. Ridge regression adds the penalty term $\alpha \left\lVert {\mathbf w} \right\rVert_{2}^{2}$ to this training error, with a hyperparameter $\alpha > 0$, while the least absolute shrinkage and selection operator adds $\alpha \left\lVert {\mathbf w} \right\rVert_{1}$ (Tibshirani, 1996).

The choice of the norm determines the learned hypothesis by empirical risk minimization. Consider the two optimization problems

$$
\min_{{\mathbf w} \in \mathbb{R}^{d}}
\left\lVert {\mathbf y} - {\mathbf X} {\mathbf w} \right\rVert_{2}
\quad \text{ and } \quad
\min_{{\mathbf w} \in \mathbb{R}^{d}}
\left\lVert {\mathbf y} - {\mathbf X} {\mathbf w} \right\rVert_{1} \text{,}
$$

which differ only in the norm of the residual. Squaring the $\ell_2$-norm does not change the set of minimizers, so minimizing the $\ell_2$-norm of the residual is equivalent to minimizing the average squared error loss. The distinction between induced and other norms matters here. An induced norm defines angles and orthogonal projections, and the squared $\ell_2$-norm of the residual is a quadratic function of ${\mathbf w}$, so the least squares problem has a closed-form solution, obtained from the normal equations (Boyd and Vandenberghe, 2004, Sect. 1.2). This is why the $\ell_2$-norm is the norm of the loss function of linear regression. Minimizing the $\ell_1$-norm of the residual is a linear program (Boyd and Vandenberghe, 2004, Sect. 6.1.1), which has no closed-form solution and is solved iteratively, as is the least absolute shrinkage and selection operator problem with its $\ell_1$-norm penalty.

Fig. 2 shows a training set of four data points, one of which is an outlier, and two linear hypotheses $h(x) = w x$ learned from it. Minimizing the squared $\ell_2$-norm of the residual gives the slope $\widehat{w} = 46/30$. This solution is moved toward the outlier because squaring makes the largest residual the main part of the training error. Minimizing the $\ell_1$-norm yields $\widehat{w} = 1$, which predicts the three remaining data points exactly and leaves a single large residual at the outlier. The hypothesis learned with the $\ell_1$-norm is robust to the outlier in the sense that one unusual data point barely moves it (Boyd and Vandenberghe, 2004, Sect. 6.1.2), while the one learned by minimizing the squared $\ell_2$-norm is not (see robustness). Both slopes are learned from the training set; which of the two hypotheses has the smaller risk on data points outside it depends on whether outliers are present there as well.

![Figure 2 of the entry norm](https://dictionaryofml.org/terms/fig/norm-fig2.svg)

*Figure 2: Two linear hypotheses $h(x) = w x$ learned from the same training set of four data points (filled circles) by minimizing a norm of the residual. The squared $\ell_2$-norm yields the solid line with slope $\widehat{w} = 46/30$, moved toward the outlier; the $\ell_1$-norm yields the dashed line with slope $\widehat{w} = 1$, which predicts the three remaining data points exactly and leaves one large residual (dotted vertical segment) at the outlier. Data generated by pythondemos/norm.py*

**See also:** vector space, metric, metric space, Euclidean space, [Hilbert space](https://dictionaryofml.org/terms/hilbertspace.html), [inner product](https://dictionaryofml.org/terms/innerproduct.html), Euclidean norm, regularizer, [linear regression](https://dictionaryofml.org/terms/linreg.html), least absolute shrinkage and selection operator.

## References

- Horn and Johnson (2013). Matrix Analysis. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781139020411>
- Axler (2024). Linear Algebra Done Right. *Springer*. <https://doi.org/10.1007/978-3-031-41026-0>
- Tibshirani (1996). Regression Shrinkage and Selection via the Lasso. *J. Roy. Statist. Soc.: Ser. B (Methodological)*. <https://doi.org/10.1111/j.2517-6161.1996.tb02080.x>
- Boyd and Vandenberghe (2004). Convex Optimization. *Cambridge Univ. Press*. <https://doi.org/10.1017/CBO9780511804441>

## Cite this entry

```bibtex
@misc{dictml_norm,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {norm},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/norm.html}
}
```


---

# online learning

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/onlinelearning.html) · [typeset PDF](https://dictionaryofml.org/terms/onlinelearning.pdf) · [LaTeX](https://dictionaryofml.org/terms/onlinelearning.tex) · [Python demo](https://dictionaryofml.org/terms/onlinelearning.py) · [notebook](https://dictionaryofml.org/terms/onlinelearning.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Online learning refers to machine learning methods that continuously update the model parameters as new data points arrive over time. This contrasts with offline methods such as linear regression, which solve empirical risk minimization once over a fixed training set. The updates are based on some form of feedback: examples are an entire loss function determined by the new data point, or a reward signal. Online gradient descent uses an entire loss function as feedback; bandit and reinforcement learning methods replace the unavailable gradient by an estimate from the feedback. The figure of merit is the regret: the accumulated loss minus that of fixed model parameters chosen in hindsight. A method is successful if its regret per round vanishes as the number of rounds grows.

## Definition

A weather station reports one temperature reading per hour, and a forecast for the next hour must be issued before that reading arrives. An offline method such as linear regression first collects readings into a training set and trains a model on it once. It then forecasts with the learned model parameters, which stay fixed from then on: while it collects, it issues no forecast, and after it has trained, it no longer learns. An online learning method instead adapts its model parameters every time a new reading arrives; Fig. 1 shows both on three days of readings.

![Figure 1 of the entry onlinelearning](https://dictionaryofml.org/terms/fig/onlinelearning-fig1.svg)

*Figure 1: Offline and online learning of a temperature forecaster on three days of hourly readings at Krems an der Donau (stems), both forecasting each hour from the readings of the three previous hours. (a) An offline method collects the readings of the first day into a training set, trains a linear regression forecaster on them once, and forecasts the next two days with the learned model parameters $\widehat{{\mathbf w}}$ held fixed; during the first day it issues no forecast. (b) An online method starts from ${\mathbf w}^{(1)} = \mathbf{0}$ at hour $0$, issues a forecast with its current model parameters ${\mathbf w}^{(t)}$ at every hour, and updates them as soon as the reading arrives; its first forecasts are far off, and within the first day it tracks the readings. Data generated by pythondemos/onlinelearning.py*

Online learning proceeds in rounds: in each round, a machine learning method commits to model parameters, receives a feedback signal from its environment, and updates (Cesa-Bianchi and Lugosi, 2006; Hazan, 2022, Sect. 1.1; Shalev-Shwartz, 2012). The updates are based on some form of feedback. For the weather station, the feedback is the new data point itself. Any new temperature reading allows to compute the squared error loss against the current model parameters, and the loss can be evaluated for any possible choice of the model parameters. The feedback is therefore an entire loss function ${\mathbf w} \mapsto L\left({\mathbf z}^{(t)},{\mathbf w} \right)$ (Hazan, 2022, Sect. 1.1). Online gradient descent is the online learning method for feedback by an entire loss function: it updates the model parameters by a gradient step on that loss function (Zinkevich, 2003).

Some online learning applications provide only a single loss or reward value as feedback, called bandit feedback in contrast to full information (Fig. 2). For example, a clinic that gives one of several treatments observes the outcome of that treatment alone, which is a multiarmed bandit (Lattimore and Szepesvári, 2020, Ch. 1). A thermostat whose heating decision changes the temperature it faces in the next hour receives a reward together with a new state, which is reinforcement learning (Sutton and Barto, 2018, Ch. 3).

![Figure 2 of the entry onlinelearning](https://dictionaryofml.org/terms/fig/onlinelearning-fig2.svg)

*Figure 2: The loss functions of five consecutive hours of the temperature stream in one plane ($f^{(1)}$ solid black, $f^{(2)}$ dashed black, $f^{(3)}$ solid gray, $f^{(4)}$ dashed gray, and $f^{(5)}$ dotted black curve), for the scalar forecaster $w$ and the squared error loss $f^{(t)}(w) = (y^{(t)} - w)^{2}$, which is smallest at the reading $y^{(t)}$ of hour $t$. With full information, the whole curve $f^{(t)}$ is revealed at hour $t$. With bandit feedback, only its value at the committed forecast $w^{(t)}$ is revealed (dots, labeled by $t$); here the forecasts are the iterates of online gradient descent, each moved toward the newest reading by a fixed fraction of the distance. Data generated by pythondemos/onlinelearning.py*

Consider model parameters ${\mathbf w}$ from a convex closed set $\mathcal{W} \subseteq \mathbb{R}^{d}$, the parameter space, and rounds at consecutive time instants $t = 1, 2, \ldots$. At time $t$, the environment fixes a function $f^{(t)}: \mathcal{W} \to \mathbb{R}$, and the current model parameters ${\mathbf w}^{(t)}$ incur the loss $f^{(t)}({\mathbf w}^{(t)})$. Only then is a feedback signal $r^{(t)}$ about $f^{(t)}$ revealed. The feedback signal is written $r^{(t)}$, in the reward notation of reinforcement learning, although with full information it is a data point and with bandit feedback a loss value. Three kinds of feedback occur.

With full information, the data point ${\mathbf z}^{(t)}$ is revealed, and it determines $f^{(t)}({\mathbf w}) = L\left({\mathbf z}^{(t)},{\mathbf w} \right)$ for every ${\mathbf w} \in \mathcal{W}$. On the temperature stream, the data point ${\mathbf z}^{(t)} = ({\mathbf x}^{(t)}, y^{(t)})$ has as features, collected in the feature vector ${\mathbf x}^{(t)}$, the readings of the previous hours, and as label the reading of hour $t$. Its loss function $f^{(t)}$ is the squared error loss of the forecast $h^{({\mathbf w})}({\mathbf x}^{(t)})$.

With bandit feedback, only the value $f^{(t)}({\mathbf w}^{(t)})$ is revealed. In a multiarmed bandit with $A$ actions, the model parameters ${\mathbf w}^{(t)} \in \Delta^{A-1}$ form the probability mass function from which the action $a^{(t)}$ is drawn. The environment fixes a loss vector $\boldsymbol{\ell}^{(t)} \in [0,1]^{A}$, whose entry $\ell^{(t)}_{a}$ is the loss of action $a$ in round $t$, so the expected loss of the draw is $f^{(t)}({\mathbf w}) = \langle \boldsymbol{\ell}^{(t)}, {\mathbf w} \rangle$. The clinic observes the single entry $\ell^{(t)}_{a^{(t)}}$, the loss of the action drawn, and no other entry.

With reinforcement learning feedback, the immediate reward $r(s^{(t)}, a^{(t)})$ of the action taken and the next state $s^{(t+1)}$ of a Markov decision process are revealed. The loss $f^{(t)}({\mathbf w})$ is the negative expected reward of the policy with model parameters ${\mathbf w}$ in the state $s^{(t)}$. That state depends on the actions of the earlier rounds, so the environment's losses depend on the method's own past.

The method then updates the model parameters using the feedback,

\begin{equation}
\tag{1}
{\mathbf w}^{(t+1)} :=
\mathcal{F}^{(t)}\big({\mathbf w}^{(t)}, r^{(t)}\big)
\text{,}
\end{equation}

with an update operator $\mathcal{F}^{(t)}$ specific to the method, the counterpart of the operator $\mathcal{F}$ of a fixed-point iteration (Fig. 3). The time index is also the iteration index: one update for any newly arriving data point. This contrasts with batch learning, which solves empirical risk minimization once over a fixed training set and then uses the learned hypothesis.

![Figure 3 of the entry onlinelearning](https://dictionaryofml.org/terms/fig/onlinelearning-fig3.svg)

*Figure 3: One round of online learning. The method commits to the model parameters ${\mathbf w}^{(t)}$ before the environment fixes the loss $f^{(t)}$ of the round, incurs $f^{(t)}({\mathbf w}^{(t)})$, receives a feedback signal $r^{(t)}$, and updates to ${\mathbf w}^{(t+1)}$ for the next round. The feedback is the data point itself (full information, as in online gradient descent), the loss of the drawn action alone (bandit), or a reward and the next state (reinforcement learning)*

The update Eq. (1) at time $t$ applies an operator that depends on the feedback signal $r^{(t)}$. By contrast, gradient descent applies a fixed gradient step operator in every time step. An online learning method is therefore not a fixed-point iteration of one fixed operator. Instead, it concatenates operators that vary over the time steps. Unlike stochastic gradient descent, which uses one data point per step to approximate the gradient of one fixed objective function, an online learning method faces a different loss function in every round.

With full information, online gradient descent uses the gradient ${\mathbf g}^{(t)} := \nabla f^{(t)}({\mathbf w}^{(t)})$ in a projected gradient step. Online mirror descent replaces the Euclidean distance of the projection by a Bregman divergence, e.g., the Kullback–Leibler divergence on the probability simplex, which gives the Hedge update of the probability mass function over the actions (Hazan, 2022, Sect. 5.4). Follow-The-Regularized-Leader commits, in every round, to a minimizer of the accumulated loss of all rounds so far plus a regularizer (Hazan, 2022, Sect. 5.2).

With bandit feedback, the gradient of $f^{(t)}$ is not available, since only one entry of $\boldsymbol{\ell}^{(t)}$ is observed. Exp3 forms an unbiased estimate of $\boldsymbol{\ell}^{(t)}$ by dividing the observed entry by the probability of the drawn action, and applies the online mirror descent update to the estimate (Auer et al., 2002, Sect. 3).

With reinforcement learning feedback, Q-learning changes one entry of a table of action values, the entry of the state visited and the action taken. The entry moves by a fixed fraction of the difference between its target, the reward received plus the discounted largest value of the next state, and its current value. This is a stochastic step of the Bellman operator, a contraction with factor $\gamma$ in the supremum norm: taking the largest value over the actions cannot increase the largest difference between two value estimates, and the discount factor scales that difference by $\gamma$. Q-learning, unlike online gradient descent, does have a fixed point interpretation (Sutton and Barto, 2018, Sect. 6.5). Policy gradient methods instead estimate the gradient of the expected reward from the rewards collected along the visited states (Sutton and Barto, 2018, Ch. 13).

The figure of merit of an online learning method is not the decrease of an objective function along the iterates, which measures the progress of gradient descent: online learning has no fixed objective function. The loss function $f^{(t)}$ of round $t$ is evaluated once, around ${\mathbf w}^{(t)}$, and the next round brings a different loss function. Instead, an online learning method is evaluated by its regret.

The regret is the excessive accumulated loss compared to that of a fixed choice ${\mathbf w} \in \mathcal{W}$ of the model parameters. This fixed choice ${\mathbf w}$ is kept for all rounds $t = 1, \ldots, T$ and can be chosen after all losses are known. After $T$ rounds, the regret relative to ${\mathbf w} \in \mathcal{W}$ is

$$
\mathrm{Regret}_{T}({\mathbf w}) :=
\sum_{t=1}^{T} f^{(t)}({\mathbf w}^{(t)})
- \sum_{t=1}^{T} f^{(t)}({\mathbf w}) \text{.}
$$

In a multiarmed bandit, the fixed ${\mathbf w}$ of interest is the probability mass function concentrated on one action. The regret is then measured against the best single action in hindsight; in reinforcement learning, against the best fixed policy (Jaksch et al., 2010, Sect. 2).

No assumption is made on how the losses are generated: the regret is defined for every sequence $f^{(1)}, f^{(2)}, \ldots$, even one chosen by an opponent who has seen ${\mathbf w}^{(t)}$. An online learning method is successful if its regret per round, $\mathrm{Regret}_{T}({\mathbf w})/T$, vanishes as $T$ grows for every ${\mathbf w} \in \mathcal{W}$ (Shalev-Shwartz, 2012, Sect. 1.1).

**Regret bounds.** The rate at which the regret per round vanishes depends on the feedback. With full information and convex losses whose gradients have norm at most $G$ on a parameter space of diameter $D$, online gradient descent with the learning rate $\eta^{(t)} = D/(G\sqrt{t})$ has $\mathrm{Regret}_{T}({\mathbf w}) \le \tfrac{3}{2} DG\sqrt{T}$ (Hazan, 2022, Thm. 3.1). When the number of rounds $T$ is known in advance, the constant learning rate $\eta = D/(G\sqrt{T})$ has $\mathrm{Regret}_{T}({\mathbf w}) \le DG\sqrt{T}$ by the same argument. No online algorithm can guarantee a regret below a constant times $DG\sqrt{T}$ for every sequence: for each algorithm there is a sequence of losses on which its regret is at least that (Hazan, 2022, Thm. 3.2).

With bandit feedback over $A$ actions and losses in $[0,1]$, the expected regret of Exp3 is at most $2\sqrt{(e-1) T A \ln A}$ (Auer et al., 2002, Cor. 3.2): observing one entry of $\boldsymbol{\ell}^{(t)}$ instead of all of them costs the factor $\sqrt{A}$. With reinforcement learning feedback in a Markov decision process with $S$ states and $A$ actions, the UCRL2 method has, with probability at least $1 - \delta$ for any $\delta \in (0, 1)$ and any $T > 1$, regret at most a constant times $S \sqrt{A T \ln(T/\delta)}$, the constant depending on how many steps it takes to reach any state from any other (Jaksch et al., 2010, Thm. 2). In each case the regret per round vanishes as $T$ grows.

Fig. 4 shows the regret per round of online gradient descent and of follow-the-leader, the Follow-The-Regularized-Leader method without its regularizer, which commits in every round to a minimizer of the accumulated loss so far, on a year of hourly temperature readings. For linear regression with the squared error loss, that minimizer is computed by the recursive least-squares update, which carries the least-squares solution from time $t - 1$ to time $t$ in closed form by the Sherman–Morrison formula for the inverse of a matrix after a rank-one change (Golub and Loan, 2013, Sect. 2.1.4). Online gradient descent is run afresh for every number of rounds $T$ shown, with the constant learning rate $\eta = D/(G\sqrt{T})$ on a parameter space of diameter $D = 4$ and with $G$ the largest gradient norm encountered, so the bound $DG/\sqrt{T}$ on its regret per round holds at every $T$ shown.

![Figure 4 of the entry onlinelearning](https://dictionaryofml.org/terms/fig/onlinelearning-fig4.svg)

*Figure 4: Online learning of a temperature forecaster for Krems an der Donau in 2024. Each new temperature reading provides a new data point, with the readings of the three previous hours as features and the squared error loss. (a) One week of readings and the forecasts issued by online gradient descent and by follow-the-leader before each reading arrived. (b) The regret per round of both methods after $T$ rounds, relative to the model parameters that are best in hindsight over the same rounds, and the bound $DG/\sqrt{T}$ of online gradient descent run with the learning rate $D/(G\sqrt{T})$ for each $T$. Data generated by pythondemos/onlinelearning.py*

Fig. 5 turns the same stream into a multiarmed bandit. The actions are five fixed forecasters, each a rule that needs no training. They forecast the previous reading, the reading of the same hour one day earlier, the mean of the three previous readings, the linear extrapolation of the two previous readings, and the mean of all readings of the year so far. The loss of an hour is the scaled absolute error of the forecaster used, and the regret is measured against the best single forecaster in hindsight.

With full information, all five errors are observed after each hour and Hedge updates the probability mass function; with bandit feedback, only the error of the forecaster used is observed and Exp3 updates the probability mass function from its estimate of the loss vector.

The regret of Hedge after $T$ rounds is at most $\sqrt{2 T \ln A}$ for losses in $[0,1]$, a weakening of the bound $\ln A / \eta + \eta T / 8$ of Cesa-Bianchi and Lugosi (2006, Thm. 2.2) at the learning rate $\eta = \sqrt{2 \ln A / T}$; for Exp3, the bound on the expected regret stated for bandit feedback applies with $A = 5$. Both regrets per round vanish, and the regret of Exp3 stays larger than that of Hedge, the increase in regret caused by bandit feedback.

![Figure 5 of the entry onlinelearning](https://dictionaryofml.org/terms/fig/onlinelearning-fig5.svg)

*Figure 5: The temperature stream as a multiarmed bandit with $A = 5$ fixed forecasters as actions. The regret per round after $T$ rounds against the best single forecaster in hindsight, for Hedge with full information (solid curve, filled circles) and for Exp3 with bandit feedback (dashed curve, filled squares), together with the bounds $\sqrt{2 \ln A / T}$ of Hedge (dotted curve) and $2\sqrt{(e-1) A \ln A / T}$ of Exp3 (dash-dotted curve). Data generated by pythondemos/onlinelearning.py*

**See also:** online gradient descent, online algorithm, regret, Hedge, Exp3, online mirror descent, mirror descent, Follow-The-Regularized-Leader, [stochastic gradient descent](https://dictionaryofml.org/terms/stochGD.html), [gradient descent](https://dictionaryofml.org/terms/gd.html), gradient step, [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), risk, multiarmed bandit, [reinforcement learning](https://dictionaryofml.org/terms/reinforcementlearning.html), Markov decision process, Q-learning, batch learning, fixed-point iteration, [linear regression](https://dictionaryofml.org/terms/linreg.html).

## References

- Cesa-Bianchi and Lugosi (2006). Prediction, Learning, and Games. *Cambridge Univ. Press*.
- Hazan (2022). Introduction to Online Convex Optimization. *MIT Press*. <https://sites.google.com/view/intro-oco/>
- Shalev-Shwartz (2012). Online Learning and Online Convex Optimization. *Foundations and Trends in Machine Learning*. <https://doi.org/10.1561/2200000018>
- Zinkevich (2003). Online Convex Programming and Generalized Infinitesimal Gradient Ascent. *Proc. 20th Int. Conf. Mach. Learn.*.
- Lattimore and Szepesvári (2020). Bandit Algorithms. *Cambridge University Press*.
- Sutton and Barto (2018). Reinforcement Learning: An Introduction. *MIT Press*.
- Auer et al. (2002). The Nonstochastic Multiarmed Bandit Problem. *SIAM J. Comput.*. <https://doi.org/10.1137/S0097539701398375>
- Jaksch et al. (2010). Near-optimal Regret Bounds for Reinforcement Learning. *Journal of Machine Learning Research*.
- Golub and Loan (2013). Matrix Computations. *The Johns Hopkins Univ. Press*.

## Cite this entry

```bibtex
@misc{dictml_onlinelearning,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {online learning},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/onlinelearning.html}
}
```


---

# overfitting

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/overfitting.html) · [typeset PDF](https://dictionaryofml.org/terms/overfitting.pdf) · [LaTeX](https://dictionaryofml.org/terms/overfitting.tex) · [Python demo](https://dictionaryofml.org/terms/overfitting.py) · [notebook](https://dictionaryofml.org/terms/overfitting.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Overfitting is the failure mode of a machine learning method that fits its training set too closely: the learned hypothesis incurs a small empirical risk on the training set but a large risk, i.e., the expected loss on data points drawn from the underlying probability distribution. Consequently, the learned hypothesis has a large generalization gap. Validation detects overfitting by comparing the training error with the validation error obtained on a validation set. Overfitting typically arises when the hypothesis space is too large for the number of data points in the training set; the Vapnik–Chervonenkis dimension and the Rademacher complexity measure that size. Regularization counteracts overfitting by pruning the model, adding a penalty term to the empirical risk, or augmenting the training set.

## Definition

An image classifier that memorizes every photograph in its training set but misclassifies images outside its training set is overfitting. In general, consider a machine learning method that uses empirical risk minimization to learn a hypothesis $\widehat{h} \in \mathcal{H}$ with minimum empirical risk $\widehat{L}\big(\widehat{h}|\mathcal{D}^{(\rm train)}\big)$ on a given training set $\mathcal{D}^{(\rm train)}$. The method overfits $\mathcal{D}^{(\rm train)}$ if the empirical risk $\widehat{L}\big(\widehat{h}|\mathcal{D}^{(\rm train)}\big)$ is small while the risk $\bar{L} \big( \widehat{h} \big)$, i.e., the expected loss on data points drawn, independently of $\mathcal{D}^{(\rm train)}$, from the underlying probability distribution, is large. Consequently, an overfitting machine learning method learns a hypothesis with a large generalization gap (Hastie et al., 2009, Sect. 7.2; Jung, 2022).

A large generalization gap alone does not imply overfitting. Consider a linear model fitted to a small training set whose labels are noisy and depend nonlinearly on the features. It can incur a large training error and a still larger risk, so the generalization gap is large while the training error is not small. That example is underfitting, the opposite failure: a hypothesis space holding no hypothesis that represents the relation between the features and the label of a data point leaves both quantities large.

Fig. 1 shows why a small training error is on its own no evidence against overfitting. Three data points lie on a straight line. None of the three curves is drawn by hand: each is a hypothesis $\widehat{h}$ delivered by empirical risk minimization with the squared error loss on those same three data points. What separates them is the hypothesis space that empirical risk minimization searched, and the three are nested, $\mathcal{H}^{(1)} \subset \mathcal{H}^{(2)} \subset \mathcal{H}^{(3)}$: $\mathcal{H}^{(1)}$ is the constants $h(x) = b$, $\mathcal{H}^{(2)}$ the linear model, whose hypotheses are $h(x) = w x + b$, and $\mathcal{H}^{(3)}$ all continuous functions on $\mathbb{R}$.

On $\mathcal{H}^{(1)}$, empirical risk minimization has a unique solution: the constant whose value is the average of the three labels. That constant meets the middle data point and misses the other two, and no member of $\mathcal{H}^{(1)}$ does better. The hypothesis space is too small here, and the method underfits. On $\mathcal{H}^{(2)}$, the solution is unique as well and attains a training error of zero; the hypothesis space matches the relation between feature and label, and the method neither underfits nor overfits. On $\mathcal{H}^{(3)}$, every continuous function through the three data points attains a training error of zero, and there are infinitely many of them: empirical risk minimization has infinitely many solutions there, and Fig. 1 draws one of them.

The training error cannot separate that curve from the solution on $\mathcal{H}^{(2)}$, the linear hypothesis that passes through the three data points, and it cannot choose among the solutions on $\mathcal{H}^{(3)}$ either. The two hypotheses agree on the training set but differ at feature values between those of the training data points, and the risk differs between them, since data points drawn from the probability distribution also fall in that region. The hypothesis space is too large here, and the method overfits.

![Figure 1 of the entry overfitting](https://dictionaryofml.org/terms/fig/overfitting-fig1.svg)

*Figure 1: The hypotheses $\widehat{h}$ that empirical risk minimization with the squared error loss delivers from three nested hypothesis spaces: the constants $\mathcal{H}^{(1)}$ (dotted line), the linear model $\mathcal{H}^{(2)}$ (dashed line), and all continuous functions $\mathcal{H}^{(3)}$ (solid curve). Filled circles are the three data points of the training set, which lie on a straight line; open circles are further data points on that same straight line, outside the training set. The same training set and the same empirical risk minimization produce all three curves, so the hypothesis space is the only thing that varies. On $\mathcal{H}^{(1)}$, the constant meets only the middle data point, which is what underfitting looks like. On $\mathcal{H}^{(2)}$, the linear hypothesis recovers the relation the data points follow. On $\mathcal{H}^{(3)}$, the training error is also zero, so the training error cannot separate the curve from the linear hypothesis. The curves differ between the training data points, which is where the open circles lie and where the risk of the two curves differs: the method overfits*

Validation detects overfitting, because the generalization gap that defines it can be estimated (Shalev-Shwartz and Ben-David, 2014, Thm. 11.1): a machine learning method that overfits delivers a small training error but a large validation error on a validation set held out from training.

Fig. 2 illustrates overfitting for polynomial regression. Each data point carries a single feature $x$ and a noisy label $y$, drawn from a probability distribution described in the caption. Polynomials of degree $p = 0, \ldots, 9$ are fitted, using empirical risk minimization with the squared error loss, to training sets of $m = 5$, $10$, and $20$ data points; each panel of Fig. 2 shows the training error and the validation error for one size. The training error decreases with increasing degree in every panel: the polynomial of degree $9$ has $10$ model parameters and interpolates the $m = 10$ data points; for $m = 5$, every degree from $4$ on interpolates; for $m = 20$, no degree shown interpolates. The validation error is obtained on a shared validation set of $100$ data points. It is in contrast smallest at a moderate degree and grows by several orders of magnitude for larger degrees. The high-degree polynomials overfit the training set, and the gap between validation error and training error estimates their large generalization gap.

Comparing the three training set sizes shows that the smaller the training set, the lower the degree at which the validation error departs from the training error. For $m = 5$, the validation error jumps by nearly three orders of magnitude already at degree $4$. That is the lowest degree whose $5$ model parameters suffice to interpolate the training set. For $m = 20$, the validation error varies by less than one order of magnitude across all degrees shown.

![Figure 2 of the entry overfitting](https://dictionaryofml.org/terms/fig/overfitting-fig2.svg)

*Figure 2: Training error and validation error of polynomial regression over the polynomial degree $p$ (dashed curve with open circles and solid curve with filled squares); panels (a), (b), and (c) correspond to a training set of $m = 5$, $10$, and $20$ data points. The feature $x$ is drawn uniformly from $[0,1]$, and the label is $y = \sin( 2 \pi x ) + \varepsilon$ with additive Gaussian noise $\varepsilon$. All three sizes share a validation set of $100$ data points drawn from the same probability distribution. The smaller the training set, the lower the degree at which overfitting sets in. Data generated by pythondemos/overfitting.py*

Overfitting typically occurs when the hypothesis space $\mathcal{H}$ is too large relative to the number of data points in the training set (as for $\mathcal{H}^{(3)}$ in Fig. 1). The Vapnik–Chervonenkis dimension (Vapnik, 2000, Sect. 3.6) of $\mathcal{H}$ and its Rademacher complexity (Shalev-Shwartz and Ben-David, 2014, Sect. 26.1) measure the size of $\mathcal{H}$ without counting model parameters. Counting model parameters is the obvious way to measure that size, but not always the right one: the hypotheses $h^{(w)}(x)
= \lceil 0.5 \sin( w x ) \rceil$ carry a single model parameter yet form a hypothesis space with infinite Vapnik–Chervonenkis dimension (Shalev-Shwartz and Ben-David, 2014, Sect. 6.3.5).

The count is exactly right, in contrast, for the classifiers $h(x)
= \operatorname{sign}\big( w_{0} + w_{1} x
+ \ldots + w_{p} x^{p} \big)$ obtained by thresholding the polynomials of maximum degree $p$ in a single feature. Their hypothesis space has Vapnik–Chervonenkis dimension $p + 1$, the number of model parameters and one more than the maximum degree. The feature map $x \mapsto (x, x^{2}, \ldots,
x^{p})$ turns these classifiers into linear classifiers on $\mathbb{R}^{p}$, whose hypothesis space has Vapnik–Chervonenkis dimension $p + 1$ (Shalev-Shwartz and Ben-David, 2014, Thm. 9.3). Conversely, any $p + 1$ data points with distinct features are shattered. The classifiers realize every possible assignment of binary labels to these data points, since a polynomial of degree at most $p$ interpolates arbitrary labels on them. These thresholded classifiers concern a different case than Fig. 2, which fits polynomials of the same degrees to a regression task.

Both measures enter the upper bounds on the generalization gap, which grow with them and shrink with the number of data points in the training set (Shalev-Shwartz and Ben-David, 2014, Sect. 26.1; Vapnik, 2000, Sect. 3.7).

Regularization counteracts overfitting in three elementary forms: pruning the model, adding a penalty term to the empirical risk, or augmenting the training set via data augmentation.

**See also:** [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), generalization gap, [generalization](https://dictionaryofml.org/terms/generalization.html), [regularization](https://dictionaryofml.org/terms/regularization.html), underfitting, validation, Vapnik–Chervonenkis dimension.

## References

- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Jung (2022). Machine Learning: The Basics. *Springer Nature*. <https://doi.org/10.1007/978-981-16-8193-6>
- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Vapnik (2000). The Nature of Statistical Learning Theory. *Springer*. <https://doi.org/10.1007/978-1-4757-3264-1>

## Cite this entry

```bibtex
@misc{dictml_overfitting,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {overfitting},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/overfitting.html}
}
```


---

# principal component analysis (PCA)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/pca.html) · [typeset PDF](https://dictionaryofml.org/terms/pca.pdf) · [LaTeX](https://dictionaryofml.org/terms/pca.tex) · [Python demo](https://dictionaryofml.org/terms/pca.py) · [notebook](https://dictionaryofml.org/terms/pca.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Principal component analysis (PCA) is the linear case of feature learning: it maps the feature vector of a data point to a vector of fewer entries by a linear map, chosen so that a second linear map rebuilds the original as well as possible. The criterion is the reconstruction error on a dataset whose labels, if it has any, are not used, since what has to be reproduced is the data point itself, which makes PCA an instance of empirical risk minimization with the squared error loss. The linear map that minimizes this error is given by the eigenvectors of the sample covariance matrix belonging to its largest eigenvalues, so PCA is computed by an eigenvalue decomposition. The error left is the sum of the eigenvalues that are dropped, which gives the error of every choice of the number of retained directions in advance. Each eigenvalue is the variance along its eigenvector, so the first direction is the one of largest variance. An autoencoder keeps the same criterion with ANNs in place of the two linear maps.

## Definition

A weather station at Krems an der Donau records eight numbers for each day of 2024: the lowest, highest and mean air temperature, precipitation, sunshine duration, humidity, air pressure and wind speed. Eight numbers per day are more than a scatterplot can show, and one of them, the mean temperature, is the midpoint of two others to within the precision of the archive. PCA replaces the eight by fewer, chosen so that a second linear map rebuilds the originals from the few with the smallest average squared error.

PCA is the linear case of feature learning, and its criterion is the reconstruction error. The encoder is a linear map

$$
{\mathbf \Phi}^{({\mathbf W})}: \mathbb{R}^{d} \rightarrow
\mathbb{R}^{d'}: {\mathbf x} \mapsto {\mathbf W} {\mathbf x}
\text{,}
$$

given by a matrix ${\mathbf W} \in \mathbb{R}^{d' \times d}$ with $d' < d$, and the decoder is a linear map ${\mathbf R} \in \mathbb{R}^{d \times d'}$ that rebuilds the raw feature vector from the *code*, the lower-dimensional representation ${\mathbf z} = {\mathbf W} {\mathbf x}$ of the data point. The usefulness of a specific ${\mathbf W}$ is the minimum reconstruction error it leaves on a dataset $\mathcal{D} = \big\{ {\mathbf x}^{(1)}, \ldots,
{\mathbf x}^{(m)} \big\}$ of feature vectors ${\mathbf x}^{(r)} \in \mathbb{R}^{d}$ with sample mean zero, the decoder being learned alongside the encoder (Bishop, 2006; Hastie et al., 2009, Sect. 14.5.1; Hotelling, 1933):

\begin{equation}
\tag{1}
\min_{{\mathbf R} \in \mathbb{R}^{d \times d'}}
\frac{1}{m} \sum_{r=1}^{m}
\left\lVert {\mathbf x}^{(r)} - {\mathbf R} {\mathbf W} {\mathbf x}^{(r)} \right\rVert_{2}^{2}
\text{.}
\end{equation}

Minimizing over ${\mathbf R}$ and ${\mathbf W}$ jointly yields the same pair; the inner minimum is taken first so that the reconstruction error becomes a loss function of ${\mathbf W}$ alone. No labels appear in the reconstruction error Eq. (1): the decoder has to reproduce the data point itself. PCA is therefore an instance of empirical risk minimization whose loss function is the squared error loss $L\left({\mathbf x},{\mathbf W} \right) = \left\lVert {\mathbf x} -
\widehat{{\mathbf R}} {\mathbf W} {\mathbf x} \right\rVert_{2}^{2}$, for the decoder $\widehat{{\mathbf R}}$ that attains the minimum.

The eigenvectors of the sample covariance matrix solve the reconstruction problem Eq. (1). The feature vectors are centered, i.e., their sample mean is zero, which is what makes the following matrix the sample covariance matrix:

$$
\widehat{\mathbf{Q}} = \frac{1}{m}
\sum_{r=1}^{m} {\mathbf x}^{(r)}
\big( {\mathbf x}^{(r)} \big)^{\top} \text{,}
$$

and it is positive semi-definite, since ${\mathbf u}^{\top} \widehat{\mathbf{Q}} {\mathbf u} = \frac{1}{m}
\sum_{r=1}^{m} \big( {\mathbf u}^{\top} {\mathbf x}^{(r)} \big)^{2} \geq 0$ for every ${\mathbf u} \in \mathbb{R}^{d}$. A symmetric matrix has an orthonormal basis of eigenvectors (Horn and Johnson, 2013; Meyer, 2000, p. 549), and a positive semi-definite one has nonnegative eigenvalues (Meyer, 2000, Sect. 7.6, p. 558), so the eigenvalue decomposition of $\widehat{\mathbf{Q}}$ is a spectral decomposition, $\widehat{\mathbf{Q}} = \sum_{j=1}^{d}
\lambda_{j} \mathbf{u}^{(j)}
\big( \mathbf{u}^{(j)} \big)^{\top}$ with eigenvalues $\lambda_{1} \geq \cdots \geq
\lambda_{d} \geq 0$ and eigenvectors $\mathbf{u}^{(1)}, \ldots, \mathbf{u}^{(d)}$ forming an orthonormal basis of $\mathbb{R}^{d}$.

Restricting the decoder ${\mathbf R}$ to orthonormal columns loses no generality, since the products ${\mathbf R} {\mathbf W}$ range over the matrices of rank at most $d'$ either way. For such an ${\mathbf R}$ the encoder attaining the minimum is the transpose ${\mathbf W} = {\mathbf R}^{\top}$ (Goodfellow et al., 2016, Sect. 2.12, Eq. (2.66); Hastie et al., 2009, Sect. 14.5.1, Eq. (14.52)): the best code holds the inner products ${\mathbf R}^{\top}
{\mathbf x}$ of ${\mathbf x}$ with the orthonormal columns of ${\mathbf R}$. The reconstruction ${\mathbf R} {\mathbf R}^{\top} {\mathbf x}$ is therefore the orthogonal projection of ${\mathbf x}$ onto the subspace they span. That projection is orthogonal to the error it leaves, and

$$
\left\lVert {\mathbf x} - {\mathbf R} {\mathbf R}^{\top} {\mathbf x} \right\rVert_{2}^{2}
= \left\lVert {\mathbf x} \right\rVert_{2}^{2} -
\left\lVert {\mathbf R}^{\top} {\mathbf x} \right\rVert_{2}^{2} \text{.}
$$

Averaging over all data points in $\mathcal{D}$ turns the reconstruction error Eq. (1) into ${\rm tr} \left(\widehat{\mathbf{Q}}\right) - {\rm tr} \left({\mathbf R}^{\top} \widehat{\mathbf{Q}} {\mathbf R}\right)$. The first trace does not depend on ${\mathbf R}$, so minimizing the error is maximizing the second over matrices with orthonormal columns. That maximum is $\sum_{j=1}^{d'} \lambda_{j}$, attained by the eigenvectors belonging to the $d'$ largest eigenvalues (Goodfellow et al., 2016, Sect. 2.12, Eq. (2.84); Horn and Johnson, 2013, Cor. 4.3.39). A matrix ${\mathbf W} = \big( \mathbf{u}^{(1)}, \ldots,
\mathbf{u}^{(d')} \big)^{\top}$ whose rows are those eigenvectors therefore minimizes the criterion Eq. (1), paired with the decoder $\widehat{{\mathbf R}} = {\mathbf W}^{\top}$. The minimum reconstruction error is $\sum_{j=d'+1}^{d}
\lambda_{j}$, the eigenvalues left out, so the spectrum of $\widehat{\mathbf{Q}}$ gives the reconstruction error of every code length in advance.

The eigenvectors $\mathbf{u}^{(1)}, \ldots,
\mathbf{u}^{(d)}$ of $\widehat{\mathbf{Q}}$ are the *principal directions* of the dataset, ordered by decreasing eigenvalue. Stacking the centered feature vectors row-wise into the feature matrix ${\mathbf X} \in
\mathbb{R}^{m \times d}$ writes the sample covariance matrix as $\widehat{\mathbf{Q}} = (1/m) {\mathbf X}^{\top} {\mathbf X}$, so they are equally the eigenvectors of ${\mathbf X}^{\top} {\mathbf X}$. The first $d'$ of them span the subspace that PCA keeps, and each feature vector is replaced by its orthogonal projection onto it. That projection is the approximation whose squared error loss the reconstruction error Eq. (1) measures. The inner products of ${\mathbf x}$ with the first $d'$ principal directions are its coordinates in that subspace, and they are the new features PCA delivers.

The eigenvalues of $\widehat{\mathbf{Q}}$ are the variances of the data along its eigenvectors, so $\mathbf{u}^{(1)}$ is the direction of largest variance. Fig. 1 shows the two directions for the days at Krems taken with two features, the lowest and the highest temperature of the day (both centered). Their eigenvalues are $130.9$ and $7.9$ squared degrees, whose square roots are the standard deviations along them.

![Figure 1 of the entry pca](https://dictionaryofml.org/terms/fig/pca-fig1.svg)

*Figure 1: Twenty of the $366$ days of 2024 at Krems an der Donau (every $19$th day), each drawn as a gray dot, a data point with the lowest and the highest temperature of the day as its two features, centered. The arrows are the eigenvectors of the sample covariance matrix of all $366$ days ($\mathbf{u}^{(1)}$ solid and $\mathbf{u}^{(2)}$ dashed arrow), each drawn two standard deviations of the data long, so the longer arrow marks the direction of larger variance. Keeping only $\mathbf{u}^{(1)}$ replaces each day by its position on that arrow. Data generated by pythondemos/pca.py*

On all eight measurements, each scaled to unit sample variance because temperatures, precipitation and pressure are measured in different units, the two largest eigenvalues carry $64\%$ of the total variance. A code of two numbers leaves a reconstruction error of $2.91$, exactly the sum of the six dropped eigenvalues. Fig. 2 plots the error left by every code length.

The code length $d'$ is a hyperparameter. Each number added to the code removes exactly the next eigenvalue from the error, and the ones removed first are the largest (see Fig. 2). No choice is optimal on the criterion itself, which falls to zero at $d' = d$, where nothing is compressed. What picks $d'$ is the use the code is put to: a scatterplot needs two, a later learning task can be given the validation error of each length to choose from, and a storage budget fixes it outright.

![Figure 2 of the entry pca](https://dictionaryofml.org/terms/fig/pca-fig2.svg)

*Figure 2: The reconstruction error left by a code of $d'$ numbers, on the eight Krems an der Donau measurements scaled to unit sample variance. It is the sum of the eigenvalues the code drops, so it falls fastest at the start and reaches zero only when nothing is compressed. Data generated by pythondemos/pca.py*

In particular, on the same eight measurements the smallest eigenvalue reveals as much as the largest. It is $6.5 \cdot 10^{-6}$ against $3.62$ for the largest, and its eigenvector loads on the three temperatures alone: the mean temperature of a day is the midpoint of its lowest and its highest, to within the $0.05$ degrees the archive records. A direction of almost no variance is a dependency among the features, and PCA finds it without a test for it. This is what makes PCA a dimensionality reduction method rather than only a feature learning one: the features it drops are the ones the others already determine.

These numbers are averages over the dataset the eigenvectors were computed from. Whether the subspace also rebuilds data points outside it is the question of generalization. The answer depends on how well $\widehat{\mathbf{Q}}$ estimates the covariance matrix of data points outside the dataset.

The same criterion compresses images. Cutting a photograph into patches of $64 \times 64$ pixels makes each patch a data point with $12288$ features, three color values per pixel, and PCA replaces those by a code of $d'$ numbers (see Fig. 3). At $d' = 5$ the mountain and the meadow are already in place while the patch boundaries still show; at $15$ the reconstruction is close to the original.

![Figure 3 of the entry pca](https://dictionaryofml.org/terms/fig/pca-fig3.svg)

*Figure 3: PCA on the $234$ patches of a photograph of the Oetscher massif, each patch a data point. (a) The original. (b), (c), (d) Every patch put through a code of $d' = 5$, $10$ and $15$ numbers and rebuilt, leaving $29\%$, $24\%$ and $21\%$ of the total variance unexplained. Data generated by pythondemos/pca.py*

Three variants of PCA change one thing each. Probabilistic principal component analysis keeps the subspace but replaces the criterion by a probabilistic model. The code is a Gaussian random variable, and the data point is its linear image plus isotropic Gaussian noise. Maximum likelihood, computed by expectation–maximization, recovers the PCA subspace as the noise variance tends to zero (Tipping and Bishop, 1999, Sect. 3.2, Eq. (7), and Sect. 3.4).

Sparse principal component analysis keeps the criterion but adds a penalty on the entries (loadings) of an eigenvector. Each direction then involves few features and can be interpreted as a named combination of them. On the eight measurements, four loadings, the three temperatures and the sunshine duration, keep $81\%$ of the variance of $\mathbf{u}^{(1)}$ (Zou et al., 2006).

Robust principal component analysis replaces the squared error loss, which a single grossly corrupted entry can dominate, by a decomposition of the data matrix into a part of low rank plus a part with few nonzero entries. A small number of faulty measurements then falls into the second part instead of tilting the directions (Candès et al., 2011).

An autoencoder keeps the reconstruction error Eq. (1) as its criterion and replaces the two linear maps by ANNs, which is the nonlinear counterpart of PCA. With a linear decoder and the squared error loss it reconstructs into the same subspace as PCA (Goodfellow et al., 2016, Sect. 14.1).

**See also:** [feature learning](https://dictionaryofml.org/terms/featlearn.html), feature transformation, spectrum, encoder, decoder, [autoencoder](https://dictionaryofml.org/terms/autoencoder.html), [dimensionality reduction](https://dictionaryofml.org/terms/dimred.html), [eigenvalue decomposition](https://dictionaryofml.org/terms/evd.html), spectral decomposition, sample covariance matrix, eigenvector, [eigenvalue](https://dictionaryofml.org/terms/eigenvalue.html), [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), subspace, probabilistic principal component analysis, sparse principal component analysis, [robust principal component analysis](https://dictionaryofml.org/terms/rpca.html).

## References

- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Hotelling (1933). Analysis of a Complex of Statistical Variables into Principal Components. *Journal of Educational Psychology*.
- Horn and Johnson (2013). Matrix Analysis. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781139020411>
- Meyer (2000). Matrix Analysis and Applied Linear Algebra. *SIAM*.
- Goodfellow et al. (2016). Deep Learning. *MIT Press*.
- Tipping and Bishop (1999). Probabilistic Principal Component Analysis. *J. Roy. Statist. Soc.: Ser. B (Statist. Methodology)*. <https://doi.org/10.1111/1467-9868.00196>
- Zou et al. (2006). Sparse Principal Component Analysis. *Journal of Computational and Graphical Statistics*. <https://doi.org/10.1198/106186006X113430>
- Candès et al. (2011). Robust Principal Component Analysis?. *Journal of the ACM*. <https://doi.org/10.1145/1970392.1970395>

## Cite this entry

```bibtex
@misc{dictml_pca,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {principal component analysis (PCA)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/pca.html}
}
```


---

# policy (reinforcement learning)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-07. [HTML](https://dictionaryofml.org/terms/policy.html) · [typeset PDF](https://dictionaryofml.org/terms/policy.pdf) · [LaTeX](https://dictionaryofml.org/terms/policy.tex) · [Python demo](https://dictionaryofml.org/terms/policy.py) · [notebook](https://dictionaryofml.org/terms/policy.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A policy in reinforcement learning is a function that maps the current state to an action. Sometimes a policy is stochastic and defined by a conditional probability distribution over the actions given the state. The policy of a reinforcement learning method can be viewed as the hypothesis of a machine learning method. It takes the features that characterize the current state of the agent as input and outputs a prediction of the best action to take. Instead of an average loss on a training set, reinforcement learning methods use the rewards observed after taking actions to guide the learning of a policy.

## Definition

A self-driving car observes its position and heading, chooses a steering direction, and is told how well it follows the road; Fig. 1 shows three time instants of it. In reinforcement learning, such an agent interacts with an environment: at each time instant $t = 1, 2, \ldots$, the agent observes the state $s_{t}$ of the environment, takes an action $a_{t}$, and receives the next state $s_{t+1}$ together with a reward $r_{t+1}$. This interaction is modeled as a Markov decision process. For the car, the state is the position and heading of the car, the action is the steering direction, and the reward tells how well the car follows the road.

In its simplest form, a policy is a function that specifies which action $a_{t}$ to take in this Markov decision process when the current state is $s_{t}$ (Sutton and Barto, 2018, Sect. 3.5). A policy differs from a plan, a fixed sequence of actions: a policy reacts to the state observed. A policy of this kind is deterministic. It is a special case of a stochastic policy, which defines a conditional probability distribution $\mathbb{P}^{(a \mid s)}$ over the actions given the state, a probability distribution from which the action is drawn.

![Figure 1 of the entry policy](https://dictionaryofml.org/terms/fig/policy-fig1.svg)

*Figure 1: A self-driving car at three time instants, between the two contours of a road. The state $s_{t}$ is the position and heading of the car; the action $a_{t}$ selected by the policy is the steering direction (arrows). At $t$ the direction is slightly off the center of the road, so the car drifts towards the outer contour and the reward $r_{t+1}$ is low. The direction at $t+1$ is better aligned with the center, so the car returns to it and the reward $r_{t+2}$ is larger*

A policy can be viewed as a hypothesis map $h({\mathbf x})$ that predicts the best action to take. The feature vector ${\mathbf x}$ fed into the hypothesis is either the state itself, when the state is fully observed, or some indirect measurements of it, when the state is only partially observed. What sets supervised learning apart from reinforcement learning is how the usefulness of a hypothesis is assessed. Instead of computing the average loss on a dataset of labeled data points, reinforcement learning methods use the expected return obtained when the policy is applied to a Markov decision process.

The return at time instant $t$ is the discounted sum of the rewards received afterwards,

$$
G_{t} := \sum_{k=0}^{\infty}
\gamma^{k} r_{t+k+1} \text{,}
$$

with a discount factor $\gamma \in [0,1]$. The sum has a finite value when $\gamma < 1$ and the rewards are bounded (Sutton and Barto, 2018, Sect. 3.3). The case $\gamma = 1$ is used only for episodes that terminate after finitely many time instants, so that the sum has finitely many nonzero terms (Sutton and Barto, 2018, Sect. 3.4). The expected return, as a function of the starting state, is the value function of the policy (Sutton and Barto, 2018, Sect. 3.5). It can be estimated by executing the policy: the agent starts from the state several times, follows the policy for a number of time instants, records the rewards, and averages the observed returns (Sutton and Barto, 2018, Ch. 5).

**See also:** action, Markov decision process, state, [reinforcement learning](https://dictionaryofml.org/terms/reinforcementlearning.html), [reward](https://dictionaryofml.org/terms/reward.html), value function, return, policy gradient, agent.

## References

- Sutton and Barto (2018). Reinforcement Learning: An Introduction. *MIT Press*.

## Cite this entry

```bibtex
@misc{dictml_policy,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {policy (reinforcement learning)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/policy.html}
}
```


---

# pretraining

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/pretraining.html) · [typeset PDF](https://dictionaryofml.org/terms/pretraining.pdf) · [LaTeX](https://dictionaryofml.org/terms/pretraining.tex) · [Python demo](https://dictionaryofml.org/terms/pretraining.py) · [notebook](https://dictionaryofml.org/terms/pretraining.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Pretraining is the training of a model on a large, generic dataset to produce model parameters that are reused as the starting point for a later learning task. The loss minimized is typically self-supervised, so no manual annotation labour is needed. For a large language model, pretraining minimizes the average next-token prediction loss over a large text corpus. While transfer learning is the reuse of what was learned on one learning task for another, pretraining is the stage that produces what is reused, with a proxy task chosen for that purpose. The reuse takes several forms, by fine-tuning all model parameters from the pretrained ones, or by training only a subset of them on top of a pretrained feature transformation.

## Definition

Before a deep net is trained to detect tumors in a few hundred annotated scans, it is trained on millions of ordinary photographs that have nothing to do with tumors (Esteva et al., 2017). That first step is pretraining: the training of a model on a large, generic dataset to produce model parameters that are reused as the starting point for a later learning task.

The loss minimized during pretraining is typically self-supervised (see self-supervised learning): the label of a data point is constructed from the raw data itself, so no manual annotation labour is needed. For a large language model, pretraining minimizes the average next-token prediction loss over a large text corpus (Brown et al., 2020). The model parameters learned during pretraining are then adapted separately for each task-specific dataset by fine-tuning (Fig. 1). In the tumor example, the task-specific data points are the annotated scans, while the generic dataset is ImageNet, a corpus of more than three million labeled images at its release (Deng et al., 2009), so that pretraining is supervised, the exception to the self-supervised rule.

The reuse takes a second form besides fine-tuning: a pretrained feature transformation $\phi \colon \mathcal{X} \to \mathbb{R}^{d}$ is concatenated with a task-specific model. In a deep net, the first layers are the pretrained $\phi$ and the final layers are trained separately for each learning task (Fig. 2) (Yosinski et al., 2014). The same $\phi$ serves every learning task, so generic features are learned once, and each later learning task starts from an informative initialization instead of from scratch (Yosinski et al., 2014).

![Figure 1 of the entry pretraining](https://dictionaryofml.org/terms/fig/pretraining-fig1.svg)

*Figure 1: Pretraining learns model parameters $\widehat{{\mathbf w}}$ from a large dataset with labels constructed from the raw data. Each later learning task fine-tunes $\widehat{{\mathbf w}}$ on its own small dataset*

![Figure 2 of the entry pretraining](https://dictionaryofml.org/terms/fig/pretraining-fig2.svg)

*Figure 2: A deep net whose first two layers are the pretrained feature transformation $\phi$ (dashed box). Pretraining learns these layers once on the large generic dataset. The output layer is trained separately on the small dataset of each learning task, on top of the same $\phi$*

**A worked example.** A piece of an aerial photograph of the Wachau, $2.4$ km by $1.2$ km, is cut into $4608$ patches of $25$ m. A parcel register labels a patch as vineyard when more than half of its pixels lie on vineyard parcels, which is the case for a third of them. The western half is the training set, the eastern half the test set. At this resolution the rows of vines are visible stripes, while the mean color of a patch tells little: vineyards, meadows and gardens are all green (Fig. 3).

![Figure 3 of the entry pretraining](https://dictionaryofml.org/terms/fig/pretraining-fig3.svg)

*Figure 3: The piece of the aerial photograph of the Wachau (left) and the parcel register that labels its patches (right): a patch counts as vineyard when more than half of its pixels lie on a vineyard parcel, shown white. The dashed line is the split into the western training set and the eastern test set. Photograph: basemap.at (CC BY 4.0); parcels: INVEKOS 2025, AMA (CC BY 3.0 AT). Data generated by pythondemos/pretraining.py*

Pretraining uses the photograph alone, never the labels. The central $8 \times 8$ pixel block of a western patch is blanked. A convolutional neural network of two convolutional layers, each with eight $3 \times 3$ filters and a rectified linear unit, is followed by a linear readout at the blanked block. It is trained to predict the mean color of the blanked pixels from their surroundings. On the eastern patches the trained network predicts the blanked color with a squared error of $0.0008$, a quarter of the error of the mean color of the rest of the patch (Fig. 4, top).

A second pretext task trains the same network to tell whether a patch was rotated by $0$, $90$, $180$ or $270$ degrees. The direction of the rows and of the shadows answers that question. The network recognizes the rotation of $37$ percent of the eastern patches, against $25$ by chance.

The two convolutional layers are then frozen: they are the feature transformation $\phi$, and their output on the original patch, averaged over a $4 \times 4$ grid, gives $128$ features. Logistic regression on these features is a linear classifier of vineyard against no vineyard. It is trained on as few as $8$ and at most $128$ labeled data points of the western half, and its accuracy is measured on the eastern half (Fig. 4, bottom). Labels are scarce on purpose: this is the regime in which pretraining decides the outcome.

With so few labels a linear classifier on the pixels themselves is prone to overfitting: its Vapnik–Chervonenkis dimension is the number of features plus one (Shalev-Shwartz and Ben-David, 2014, Thm. 9.3), and the number of labels that generalization requires grows in proportion to it (Shalev-Shwartz and Ben-David, 2014, Thm. 6.8). It has $768$ features, the patch pooled to $16 \times 16$ pixels in three colors, and at most $128$ patches to learn them from, and its accuracy stays below the majority baseline of $0.69$ until $128$ labels are available, where it reaches $0.72$. The mean color of a patch stays at that baseline too, $0.67$ with $128$ labels. The control that every pretraining claim needs is the same two layers with their random initial weights, never trained: they reach $0.76$ with $128$ labels, since random convolutional filters with pooling already respond to texture (Saxe et al., 2011). The color-pretrained layers exceed that control from $32$ labels on, and the rotation-pretrained layers exceed it at every number of labels, reaching $0.82$ with $128$.

Two conclusions follow. Pretraining helps when the labeled dataset is too small to learn what the pretext task already taught the layers. The pretext task matters too. A task that needs the direction of the stripes yields features that a classifier of vineyards needs, while predicting a color from its surroundings is largely solved by averaging. The numbers are computed by pythondemos/pretraining.py.

![Figure 4 of the entry pretraining](https://dictionaryofml.org/terms/fig/pretraining-fig4.svg)

*Figure 4: Top: eight eastern patches of $25$ m with the blanked center (white), the center color predicted by the pretrained convolutional neural network, and the original. Bottom: accuracy of a linear classifier of vineyard patches on the eastern half against the number of labeled western patches, for five feature sets: the mean color (dotted gray curve, filled triangles), the pooled patch (dashed gray curve, filled squares), untrained layers (solid gray curve, filled circles), layers pretrained to predict the blanked color (solid black curve, filled diamonds), layers pretrained to recognize the rotation (solid black curve, inverted filled triangles), and the majority baseline (dotted black line). Photograph: basemap.at (CC BY 4.0); parcels: INVEKOS 2025, AMA (CC BY 3.0 AT). Data generated by pythondemos/pretraining.py*

**Synonyms:** pre-training.

**See also:** [transfer learning](https://dictionaryofml.org/terms/transferlearning.html), [fine-tuning](https://dictionaryofml.org/terms/finetuning.html), [self-supervised learning](https://dictionaryofml.org/terms/selfsupervisedlearning.html), foundation model, multitask learning, deep net.

## References

- Esteva et al. (2017). Dermatologist-level classification of skin cancer with deep neural networks. *Nature*.
- Brown et al. (2020). Language Models are Few-Shot Learners. *Adv. Neural Inf. Process. Syst.*. <https://proceedings.neurips.cc/paper_files/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html>
- Deng et al. (2009). ImageNet: A large-scale hierarchical image database. *2009 IEEE Conference on Computer Vision and Pattern Recognition*. <https://doi.org/10.1109/CVPR.2009.5206848>
- Yosinski et al. (2014). How Transferable Are Features in Deep Neural Networks?. *Adv. Neural Inf. Process. Syst. (NIPS)*.
- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Saxe et al. (2011). On Random Weights and Unsupervised Feature Learning. *Proceedings of the 28th International Conference on Machine Learning (ICML)*.

## Cite this entry

```bibtex
@misc{dictml_pretraining,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {pretraining},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/pretraining.html}
}
```


---

# probability distribution

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/probdist.html) · [typeset PDF](https://dictionaryofml.org/terms/probdist.pdf) · [LaTeX](https://dictionaryofml.org/terms/probdist.tex) · [Python demo](https://dictionaryofml.org/terms/probdist.py) · [notebook](https://dictionaryofml.org/terms/probdist.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A probability distribution of a random variable is the probability measure that the random variable induces on the set of its values: it assigns a probability to every measurable set of values. A machine learning method reads in a dataset and delivers an output, such as a learned hypothesis or the prediction for a given data point, and should work well for any dataset that could arise in the machine learning application at hand. The independent and identically distributed assumption models such a dataset as realizations of independent and identically distributed RVs with a common probability distribution, which is postulated from domain knowledge or estimated from a dataset. The probability distribution of a binary random variable is fully specified by a single probability, that of a discrete random variable by a probability mass function, and that of a real-valued random variable with a probability density function by that probability density function.

## Definition

Consider a machine learning method that takes as input a dataset $\mathcal{D} = \big\{ {\mathbf z}^{(1)}, \,\ldots, \,{\mathbf z}^{(m)} \big\}$, each data point a numeric feature paired with a numeric label, and delivers an output, such as a learned hypothesis $\widehat{h}$ or the prediction for a given data point. Feeding a different dataset into the same method can deliver a different output. In general, the method should work well for any dataset that could arise in the machine learning application at hand. The independent and identically distributed assumption models such a dataset as independent and identically distributed RVs ${\mathbf z}^{(1)}, \,\ldots, \,{\mathbf z}^{(m)} \sim
\mathbb{P}$ with a common probability distribution $\mathbb{P}$ (see Fig. 1). The outputs obtained from different such datasets are themselves realizations of a random variable, whose probability distribution depends on $\mathbb{P}$ and on the method. How much the outputs spread, i.e., how much they differ from one another, is therefore determined by $\mathbb{P}$ and the method.

![Figure 1 of the entry probdist](https://dictionaryofml.org/terms/fig/probdist-fig1.svg)

*Figure 1: Three datasets of realizations of independent and identically distributed RVs drawn from a common probability distribution $\mathbb{P}$, and the three outputs of the same machine learning method $\mathcal{A}$. These three outputs are realizations of another random variable $\widehat{h}$; the probability distribution of $\widehat{h}$ depends on $\mathbb{P}$ and the machine learning method. How much the three outputs differ from one another is therefore determined by $\mathbb{P}$ and the machine learning method*

Formally, a random variable is a measurable function ${\mathbf z}: \Omega \to \mathcal{Z}$ from a probability space $(\Omega,\, \Sigma,\, \mathbb{P})$ to a measurable space $\mathcal{Z}$, the set of values that ${\mathbf z}$ can take. The probability distribution of ${\mathbf z}$ is the probability measure $\mathbb{P}^{({\mathbf z})}$ that ${\mathbf z}$ induces on $\mathcal{Z}$. This probability measure assigns to every measurable subset $\mathcal{B} \subseteq \mathcal{Z}$ the probability

$$
\mathbb{P}^{({\mathbf z})}(\mathcal{B}) :=
\mathbb{P}\left({\mathbf z} \in \mathcal{B}\right) =
\mathbb{P}\big( \{ \omega \in \Omega :
{\mathbf z}(\omega) \in \mathcal{B} \} \big)
\text{,}
$$

the probability of the event that ${\mathbf z}$ takes a value in $\mathcal{B}$ (see Fig. 2; Billingsley, 1986, Sect. 13, Eq. (13.7), and Sect. 14, Eq. (14.1); Gray, 2009; Kallenberg, 1997). That event is the preimage ${\mathbf z}^{-1}(\mathcal{B}) = \{ \omega \in
\Omega : {\mathbf z}(\omega) \in \mathcal{B} \}$, the set of outcomes $\omega$ of the sample space $\Omega$ that ${\mathbf z}$ sends into $\mathcal{B}$. Measurability of ${\mathbf z}$ guarantees that ${\mathbf z}^{-1}(\mathcal{B})$ belongs to the σ-algebra $\Sigma$, so that $\mathbb{P}$ assigns ${\mathbf z}^{-1}(\mathcal{B})$ a probability. The superscript of $\mathbb{P}^{({\mathbf z})}$ names the random variable whose distribution is meant, as in $\mathbb{P}^{({\mathbf x},y)}$ for a data point with features ${\mathbf x}$ and label $y$. When only one random variable is in play, the superscript is dropped, and ${\mathbf z} \sim \mathbb{P}$ states that $\mathbb{P}$ is the probability distribution of ${\mathbf z}$.

![Figure 2 of the entry probdist](https://dictionaryofml.org/terms/fig/probdist-fig2.svg)

*Figure 2: A random variable ${\mathbf z}$ maps each outcome $\omega$ of the sample space $\Omega$ to a value ${\mathbf z}(\omega)$ in the value space $\mathcal{Z}$. Its probability distribution $\mathbb{P}^{({\mathbf z})}$ assigns to a measurable set $\mathcal{B} \subseteq \mathcal{Z}$ (right, shaded) the probability that $\mathbb{P}$ assigns to the preimage ${\mathbf z}^{-1}(\mathcal{B}) \subseteq \Omega$ (left, shaded), the event that ${\mathbf z}$ takes a value in $\mathcal{B}$*

Under the independent and identically distributed assumption, the probability distribution $\mathbb{P}$ of the data points determines how likely each dataset is: a dataset of realizations of independent and identically distributed RVs is likely when its data points lie where $\mathbb{P}$ assigns large probability. This distribution is generally unknown; only the dataset is observed. In practice, $\mathbb{P}$ is postulated based on domain knowledge or estimated from a dataset, e.g., by its empirical distribution or histogram, or by maximum likelihood within a parametric family. A parametric family is a set of probability distributions indexed by a few numbers, such as the normal distributions indexed by mean and variance.

A probability mass function or a probability density function is a compact way of specifying a probability distribution. The probability distribution of a binary random variable $y \in \{0,1\}$ is fully specified by the single probability $\mathbb{P}\left(y = 0\right)$, since $\mathbb{P}\left(y=1\right) = 1-\mathbb{P}\left(y=0\right)$; more generally, the probability distribution of a discrete random variable is specified by its probability mass function. The probability distribution of a real-valued random variable $x \in \mathbb{R}$ that has a probability density function $p^{(x)}\left(\cdot\right)$ is specified by that probability density function. The probability of a value in an interval $[a,b]$ is the integral

$$
\mathbb{P}\left( x \in [a,b] \right) = \int_{a}^{b}
p^{(x)}\left(x\right) \, \mathrm{d}x \text{,}
$$

which is approximately $p^{(x)}\left(a\right) \, |b-a|$ for a short interval $[a,b]$.

Because the probability density function gives the probability of a short interval around each value, it can be drawn. For data points ${\mathbf z} = (x, y)$ with a single numeric feature $x$ and a numeric label $y$, a probability distribution can be visualized in the plane $\mathbb{R}^{2}$. Fig. 3 shows one as a grayscale over the feature–label plane: the darker a region, the larger its probability density function, i.e., the more likely a data point drawn from $\mathbb{P}$ falls in it. The depicted probability density function is a weighted sum of two Gaussian pdfs, i.e., a Gaussian mixture model. Paralleling Fig. 1, three datasets $\mathcal{D}^{(1)}, \mathcal{D}^{(2)}, \mathcal{D}^{(3)}$, each consisting of $m = 20$ realizations of independent and identically distributed RVs with this distribution, are shown together with the three hypotheses $\widehat{h}^{(1)},
\widehat{h}^{(2)}, \widehat{h}^{(3)}$ learned from them by the same polynomial regression method. As shown in Fig. 3, the three learned hypotheses differ most where the probability density function is small.

![Figure 3 of the entry probdist](https://dictionaryofml.org/terms/fig/probdist-fig3.svg)

*Figure 3: A probability distribution over data points ${\mathbf z} = (x, y)$, drawn as a grayscale: the darker a region, the larger its probability density function. The probability density function is a Gaussian mixture model with two components. Paralleling Fig. 1, three datasets $\mathcal{D}^{(1)}$ (filled circles), $\mathcal{D}^{(2)}$ (open squares), and $\mathcal{D}^{(3)}$ (filled triangles), each of $m = 20$ realizations of independent and identically distributed RVs with this distribution, yield the three hypotheses $\widehat{h}^{(1)}$ (solid curve), $\widehat{h}^{(2)}$ (dashed curve), and $\widehat{h}^{(3)}$ (dotted curve), learned by the same polynomial regression method. The hypotheses differ most where the probability density function is small. Data generated by pythondemos/probdist.py*

The example also illustrates what empirical risk minimization can determine from a dataset. Empirical risk minimization chooses the hypothesis by its average loss over the dataset. It therefore gives little weight to a region in which the dataset contains few or no data points. Consequently, empirical risk minimization does not distinguish between hypotheses that differ only there and have nearly the same training error.

**Synonyms:** distribution, law.

**See also:** [independent and identically distributed](https://dictionaryofml.org/terms/iid.html), [independent and identically distributed assumption](https://dictionaryofml.org/terms/iidasspt.html), realization, [random variable](https://dictionaryofml.org/terms/rv.html), probability, measurable, probability density function, probability mass function, discrete random variable, probability measure, probability space, empirical distribution, expectation, [generalization](https://dictionaryofml.org/terms/generalization.html).

## References

- Billingsley (1986). Probability and Measure. *Wiley*.
- Gray (2009). Probability, Random Processes, and Ergodic Properties. *Springer Science+Business Media*. <https://doi.org/10.1007/978-1-4419-1090-5>
- Kallenberg (1997). Foundations of Modern Probability. *Springer-Verlag*.

## Cite this entry

```bibtex
@misc{dictml_probdist,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {probability distribution},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/probdist.html}
}
```


---

# projection

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/projection.html) · [typeset PDF](https://dictionaryofml.org/terms/projection.pdf) · [LaTeX](https://dictionaryofml.org/terms/projection.tex) · [Python demo](https://dictionaryofml.org/terms/projection.py) · [notebook](https://dictionaryofml.org/terms/projection.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

The projection of a vector onto a closed non-empty subset of a Euclidean space is a point in the subset that is closest to the vector in the Euclidean norm. If the subset is convex, this closest point is unique and the solution of a convex optimization problem. If the subset is a subspace, the projection is a linear map, namely the orthogonal projection onto the subspace. Every projection is idempotent, and a linear map of a Hilbert space into itself is an orthogonal projection if and only if the map is idempotent and self-adjoint. In machine learning, projections enforce constraints on model parameters during training: projected gradient descent alternates gradient steps with projections onto the constraint set, e.g., the $\ell_{1}$-ball in an equivalent formulation of the least absolute shrinkage and selection operator.

## Definition

The least absolute shrinkage and selection operator learns a linear model whose model parameters ${\mathbf w}$ must keep an $\ell_{1}$-norm below a prescribed bound, and a gradient step can move ${\mathbf w}$ outside this set. The result must then be replaced by a closest admissible vector, and training any machine learning model under a constraint set faces the same replacement at every step. The projection formalizes this replacement.

Consider a non-empty closed set $\mathcal{W} \subseteq \mathbb{R}^{d}$ in the $d$-dimensional Euclidean space. The projection $P_{\mathcal{W}}\big( {\mathbf w}\big)$ of a vector ${\mathbf w} \in \mathbb{R}^{d}$ onto $\mathcal{W}$ is

$$
P_{\mathcal{W}}\big( {\mathbf w}\big) = \mathop{\mathrm{arg\,min}}_{{\mathbf w}' \in \mathcal{W}} \left\lVert {\mathbf w} - {\mathbf w}' \right\rVert_{2}\text{,}
$$

a vector in $\mathcal{W}$ closest to ${\mathbf w}$ in the Euclidean norm. The minimum exists for every non-empty closed set $\mathcal{W}$ (Boyd and Vandenberghe, 2004, Sect. 8.1), but several closest vectors can exist. For the two-point set $\mathcal{W} = \big\{ (-1,0)^{\top}, (1,0)^{\top} \big\}$, both elements are closest to $(0,0.7)^{\top}$, and likewise to every other point on the vertical axis, which consists of exactly the points equidistant from the two elements. If $\mathcal{W}$ is closed and convex, the closest point in the Euclidean norm is unique (Boyd and Vandenberghe, 2004, Sect. 8.1).

The construction extends beyond the Euclidean space: a non-empty closed convex subset of a Hilbert space contains, for each point of the space, a unique closest point (Bauschke and Combettes, 2011, Thm. 3.14). If $\mathcal{W}$ is a subspace, that is, a subset that contains all sums and scalar multiples of its elements (such as a line through the origin), the map ${\mathbf w} \mapsto P_{\mathcal{W}}\big( {\mathbf w}\big)$ is a linear map: it is the orthogonal projection onto $\mathcal{W}$ (Axler, 2024, Thm. 6.57 and Thm. 6.61; Bauschke and Combettes, 2011, Cor. 3.22).

Every projection is idempotent: the vector $P_{\mathcal{W}}\big( {\mathbf w}\big)$ lies in $\mathcal{W}$, so its closest point in $\mathcal{W}$ is itself and $P_{\mathcal{W}}\big( P_{\mathcal{W}}\big( {\mathbf w}\big)\big) = P_{\mathcal{W}}\big( {\mathbf w}\big)$. A linear map $P$ of a Hilbert space into itself is the orthogonal projection onto the range of $P$ if and only if $P$ is idempotent and self-adjoint, i.e., $\langle P{\mathbf u}, {\mathbf v} \rangle = \langle {\mathbf u}, P{\mathbf v} \rangle$ for all ${\mathbf u}, {\mathbf v}$ (Axler, 2024, Sect. 6C, Exercise 9, and Sect. 7A, Exercise 20; Rudin, 1991, Thm. 12.14). For a matrix $\mathbf{P} \in \mathbb{R}^{d \times d}$ acting on the Euclidean space, idempotence is the condition $\mathbf{P}^{2} = \mathbf{P}$ and self-adjointness the condition $\mathbf{P}^{\top} = \mathbf{P}$. Idempotence alone is not sufficient for a linear map to be an orthogonal projection: an idempotent linear map that is not self-adjoint is an oblique projection (Meyer, 2000, Sect. 5.9). For example, the matrix $\mathbf{P} = \begin{pmatrix} 1 & 1 \\ 0 & 0 \end{pmatrix}$ on $\mathbb{R}^{2}$ satisfies $\mathbf{P}^{2} = \mathbf{P}$ but $\mathbf{P}^{\top} \neq \mathbf{P}$. Its range is the horizontal axis, and it maps ${\mathbf w} = (0,1)^{\top}$ to $\mathbf{P} {\mathbf w} = (1,0)^{\top}$, while the point of the axis closest to ${\mathbf w}$ is the origin $(0,0)^{\top}$.

In machine learning, projections enforce constraints on model parameters during training. Linear regression with the constraint $\left\lVert {\mathbf w} \right\rVert_{1} \leq \tau$, for a prescribed constant $\tau > 0$, is an equivalent formulation of the least absolute shrinkage and selection operator, for a value of its penalty hyperparameter that depends on $\tau$ and on the training set (Hastie et al., 2009, Sect. 3.4.2; Tibshirani, 1996). This constrained problem can be solved by projected gradient descent (Beck, 2017, Sect. 10.2). Each iteration takes a gradient step on the training error and then projects the result onto the $\ell_{1}$-ball $\mathcal{W} = \big\{ {\mathbf w}' \in \mathbb{R}^{d} : \left\lVert {\mathbf w}' \right\rVert_{1} \leq \tau \big\}$. The projection step is what enforces the constraint: without it, basic gradient descent on the training error can produce iterates with $\left\lVert {\mathbf w} \right\rVert_{1} > \tau$, and the result solves the unconstrained problem instead of the constrained one. Since $\mathcal{W}$ is not a subspace (scaling the vector $(\tau,0)^{\top} \in \mathcal{W}$ by $2$ yields a vector outside the ball), this projection is not a linear map but amounts to solving a convex optimization problem (Boyd and Vandenberghe, 2004, Sect. 8.1.1) (see Fig. 1). Such a projection is distinct from the proximal operator: the proximal operator minimizes, over all of $\mathbb{R}^{d}$, the sum of a convex function $f({\mathbf w}')$ and half the squared Euclidean distance to ${\mathbf w}$, while the projection minimizes the distance alone, over the given set $\mathcal{W}$.

![Figure 1 of the entry projection](https://dictionaryofml.org/terms/fig/projection-fig1.svg)

*Figure 1: The projection $P_{\mathcal{W}}\big( {\mathbf w}\big)$ is the point of the $\ell_{1}$-ball $\mathcal{W} = \big\{ {\mathbf w}' \in \mathbb{R}^{2} : \left\lVert {\mathbf w}' \right\rVert_{1} \leq \tau \big\}$ closest to a vector ${\mathbf w} \in \mathbb{R}^{2}$ in the Euclidean norm; the small square marks the right angle between the segment from ${\mathbf w}$ to $P_{\mathcal{W}}\big( {\mathbf w}\big)$ and the edge of the ball containing $P_{\mathcal{W}}\big( {\mathbf w}\big)$*

**See also:** Euclidean space, vector, minimum, [convex](https://dictionaryofml.org/terms/convex.html), orthogonal projection, proximal operator, projected gradient descent.

## References

- Boyd and Vandenberghe (2004). Convex Optimization. *Cambridge Univ. Press*. <https://doi.org/10.1017/CBO9780511804441>
- Bauschke and Combettes (2011). Convex Analysis and Monotone Operator Theory in Hilbert Spaces. *Springer Science+Business Media*. <https://doi.org/10.1007/978-1-4419-9467-7>
- Axler (2024). Linear Algebra Done Right. *Springer*. <https://doi.org/10.1007/978-3-031-41026-0>
- Rudin (1991). Functional Analysis. *McGraw-Hill*.
- Meyer (2000). Matrix Analysis and Applied Linear Algebra. *SIAM*.
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Tibshirani (1996). Regression Shrinkage and Selection via the Lasso. *J. Roy. Statist. Soc.: Ser. B (Methodological)*. <https://doi.org/10.1111/j.2517-6161.1996.tb02080.x>
- Beck (2017). First-Order Methods in Optimization. *SIAM-Society for Industrial and Applied Mathematics*.

## Cite this entry

```bibtex
@misc{dictml_projection,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {projection},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/projection.html}
}
```


---

# random experiment

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/randomexperiment.html) · [typeset PDF](https://dictionaryofml.org/terms/randomexperiment.pdf) · [LaTeX](https://dictionaryofml.org/terms/randomexperiment.tex) · [Python demo](https://dictionaryofml.org/terms/randomexperiment.py) · [notebook](https://dictionaryofml.org/terms/randomexperiment.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A random experiment is a physical or computational process that produces an outcome $\omega$ from a known set of possibilities, the sample space of the experiment. Probability theory represents the process by a probability space $(\Omega,\, \Sigma,\, \mathbb{P})$, where the σ-algebra $\Sigma$ collects the measurable events and the probability measure $\mathbb{P}\left(\cdot\right)$ assigns each event a probability. This representation is a probabilistic model of the process, not a property of it. A measurement of the outcome is represented by a random variable, a measurable function $x : \Omega \to \mathcal{X}$, and the measured value $x(\omega)$ is a realization. Repeating the experiment $m$ times is itself a random experiment on the product sample space, and the measurements of the runs are modeled as independent and identically distributed RVs. In machine learning, the collection of a training set, the draw of a subset of it by stochastic gradient descent, and the noise added for differential privacy are random experiments.

## Definition

The weather of a day at Krems an der Donau is a random experiment: the day's maximum temperature is not known in advance, but its range is. A random experiment is a physical or computational process that produces an outcome $\omega$ from a set $\Omega$ of possibilities, the sample space of the experiment (Bertsekas and Tsitsiklis, 2008, Sect. 1.2; Billingsley, 1986). The outcome is uncertain, and the experiment can be repeated under identical conditions. Fig. 1 shows the minimum and the maximum temperature of every day of January 2023 at the weather station Krems an der Donau: the weather of a day is a random experiment, and the two temperatures are measurements of its outcome.

![Figure 1 of the entry randomexperiment](https://dictionaryofml.org/terms/fig/randomexperiment-fig1.svg)

*Figure 1: The maximum (solid stems, filled circles) and the minimum temperature (dotted stems, open circles) of every day of January 2023 at the weather station Krems an der Donau. The weather of a day is a random experiment, one experiment per day, and the two temperatures are measurements of its outcome. Read the other way, the whole month is one random experiment whose outcome is all $31$ days together. Data generated by pythondemos/randomexperiment.py*

Rolling a die, reading the maximum temperature of a day at Krems, or drawing a handful of data points from a training set for one iteration of stochastic gradient descent: each of these is a random experiment.

Probability theory represents a random experiment by a probability space $(\Omega,\, \Sigma,\, \mathbb{P})$ (Bertsekas and Tsitsiklis, 2008, Sect. 1.2; Billingsley, 1986). The sample space $\Omega$ collects the possible outcomes. The σ-algebra $\Sigma$ collects the events, those subsets $\mathcal{A} \subseteq \Omega$ to which a probability is assigned. The probability measure $\mathbb{P}\left(\cdot\right)$ assigns to each event $\mathcal{A} \in
\Sigma$ a number $\mathbb{P}\left(\mathcal{A}\right) \in [0,1]$ such that $\mathbb{P}\left(\Omega\right) = 1$ and the probability of a countable union of pairwise disjoint events is the sum of their probabilities. Choosing $\Omega$, $\Sigma$ and $\mathbb{P}\left(\cdot\right)$ is a modeling step: the probability measure is a property of the probabilistic model, not of the process. For a die, the sample space is $\{1, \ldots, 6\}$ and the uniform probability measure is the model of a fair die; whether a given die is fair is a question about the process, answered by the relative frequencies of its outcomes in many rolls.

What is observed of an outcome is a measurement, a function $x : \Omega \to \mathcal{X}$ of the outcome. Such a measurable function is a random variable, and the value $x(\omega)$ it takes for the actual outcome $\omega$ is a realization of it (Billingsley, 1986; Gray, 2009). The temperature read at 13:00 is a random variable on the weather of a day, and the number read on one day is a realization. The probability measure on $\Omega$ induces the probability distribution of the random variable, $\mathbb{P}^{(x)}(\mathcal{B}) = \mathbb{P}\left(x \in
\mathcal{B}\right)$ for every measurable set $\mathcal{B} \subseteq
\mathcal{X}$ (Billingsley, 1986, Sect. 20). Machine learning methods work with the RVs and their probability distributions; the sample space behind them rarely needs to be written down. Fig. 2 depicts the two layers, the process and its mathematical representation.

![Figure 2 of the entry randomexperiment](https://dictionaryofml.org/terms/fig/randomexperiment-fig2.svg)

*Figure 2: A random experiment and its mathematical representation. The process produces an outcome $\omega$, of which a measurement $x(\omega)$ is observed. Probability theory represents the process by a probability space $(\Omega,\, \Sigma,\, \mathbb{P})$ and the measurement by a random variable $x$, a measurable function on $\Omega$ whose probability distribution $\mathbb{P}^{(x)}$ is induced by $\mathbb{P}\left(\cdot\right)$*

Repeating a random experiment $m$ times defines a new random experiment whose outcomes are sequences $(\omega^{(1)}, \ldots, \omega^{(m)})$ of outcomes of the original one (Fig. 3). Its sample space is the product $\Omega \times \cdots \times \Omega$, and repetition under identical conditions, with no run influencing another, is represented by the product probability measure: the measurements $x^{(1)}, \ldots, x^{(m)}$ of the runs are then independent and identically distributed RVs (Bertsekas and Tsitsiklis, 2008, Sect. 1.5). While the outcome of a single run is uncertain, the long-run behavior of repeated runs becomes predictable: the law of large numbers and the central limit theorem make this precise (Bertsekas and Tsitsiklis, 2008, Sects. 5.2 and 5.4).

![Figure 3 of the entry randomexperiment](https://dictionaryofml.org/terms/fig/randomexperiment-fig3.svg)

*Figure 3: Repeating a random experiment $m$ times yields another random experiment whose outcomes are sequences $(\omega^{(1)}, \,\omega^{(2)}, \,\dots, \,\omega^{(m)}) \in \Omega\times\ldots\times \Omega$. One random experiment arising in many machine learning applications is the gathering of a training set ${\mathbf z}^{(1)},\,\ldots,\,{\mathbf z}^{(m)}$, whose data points are RVs on the repeated runs*

Random experiments arise in machine learning in three places:

- Data collection: the data points of a training set are modeled as RVs, i.e., as functions of the outcome $\omega \in \Omega$ of a random experiment, and the independent and identically distributed assumption models their collection as the repetition of one experiment. This is what makes the risk of a hypothesis, an expectation under $\mathbb{P}^{({\mathbf z})}$, a meaningful target for empirical risk minimization.
- Stochastic algorithms: stochastic gradient descent runs a random experiment at each iteration, the draw of a subset of the training set; the bootstrap draws resamples of the training set.
- Privacy protection methods run a random experiment to perturb the outputs of a machine learning method, which is what differential privacy requires.

**See also:** outcome, sample space, event, [random variable](https://dictionaryofml.org/terms/rv.html), realization, probability, probability space, probability measure, [probability distribution](https://dictionaryofml.org/terms/probdist.html), probabilistic model, [independent and identically distributed](https://dictionaryofml.org/terms/iid.html), law of large numbers, central limit theorem.

## References

- Bertsekas and Tsitsiklis (2008). Introduction to Probability. *Athena Scientific*.
- Billingsley (1986). Probability and Measure. *Wiley*.
- Gray (2009). Probability, Random Processes, and Ergodic Properties. *Springer Science+Business Media*. <https://doi.org/10.1007/978-1-4419-1090-5>

## Cite this entry

```bibtex
@misc{dictml_randomexperiment,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {random experiment},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/randomexperiment.html}
}
```


---

# random forest

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/randomforest.html) · [typeset PDF](https://dictionaryofml.org/terms/randomforest.pdf) · [LaTeX](https://dictionaryofml.org/terms/randomforest.tex) · [Python demo](https://dictionaryofml.org/terms/randomforest.py) · [notebook](https://dictionaryofml.org/terms/randomforest.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A random forest is the instance of bagging whose base learners learn decision trees. Each decision tree is trained on randomly selected data points of the training set, and each base learner also uses only a randomly selected subset of the features. The predictions of the base learners are aggregated by a majority vote in classification or by averaging in regression. The random selection of data points and of features aims at a decorrelation, so that aggregating reduces the variance of the returned prediction. Aggregation leaves the bias unchanged, and adding base learners does not cause overfitting. A random forest also yields a feature importance score per feature: the fraction of the overall splits that use that feature.

## Definition

A bank must decide, for each loan application, whether the applicant is creditworthy. A random forest answers with the majority vote of many different decision trees. A random forest is the instance of bagging whose base learners learn decision trees. As in bagging, each decision tree is trained on randomly selected data points of the training set. Besides the random selection of data points, each base learner also uses only a randomly selected subset of the features of the original data points (Breiman, 2001). The predictions of the base learners are aggregated by a majority vote in classification or by averaging in regression (Fig. 2). The scatterplot of Fig. 1 shows a regression example: each data point is one of the 366 days of 2020 at the weather station Krems an der Donau, its feature the minimum and its label the maximum air temperature of the day. Three decision trees, each trained on its own random selection of the days, yield three different piecewise constant hypotheses; the random forest delivers their average.

![Figure 1 of the entry randomforest](https://dictionaryofml.org/terms/fig/randomforest-fig1.svg)

*Figure 1: Scatterplot of $20$ of the $366$ days of 2020 at the weather station Krems an der Donau (every $19$th day), with the minimum temperature of a day as feature $x$ and its maximum temperature as label $y$. Three decision trees $\widehat{h}^{(1)}, \widehat{h}^{(2)}, \widehat{h}^{(3)}$, each trained on its own random selection of the days (dotted, dashed and dash-dotted step curves). The thick solid step curve is the random forest $\widehat{h}$, which delivers the average of the three. Data generated by pythondemos/randomforest.py*

This construction works because averaging many high-variance predictions yields one of lower variance, provided the errors of the individual predictions are weakly correlated. A deep decision tree is well suited as a base learner here: it can fit the training set closely (small bias), but its prediction varies strongly with the training set (large variance). Averaging reduces this variance and leaves the bias unchanged, since the trees are identically distributed (Hastie et al., 2009, Sect. 15.2). Adding trees does not cause overfitting: as their number grows, the averaged prediction converges to its expectation over the random selections, and only this limit can overfit (Hastie et al., 2009, Sect. 15.3.4). Consider $M$ identically distributed predictions, each with variance $\sigma^{2}$ and with pairwise correlation $\rho$. The averaged prediction has variance

$$
\rho \sigma^{2} + \frac{1-\rho}{M}\, \sigma^{2}
\text{,}
$$

which decreases with the number $M$ of trees toward the lower bound $\rho \sigma^{2}$ set by the correlation (Hastie et al., 2009, Eq. (15.1)). The random selections are drawn independently of each other once the training set is fixed, which is what makes the predictions identically distributed. They are not independent when the underlying dataset is itself a random variable: every decision tree is grown from the same training set, and that shared dependence results in a nonzero correlation $\rho$ in general (Hastie et al., 2009, Sect. 15.4.1, Eq. (15.6)).

The random selection of data points and the random selection of features both aim at a decorrelation of the predictions that the base learners deliver for a feature vector ${\mathbf x}$ (Hastie et al., 2009, Sect. 15.2). While withholding features lowers the correlation $\rho$ between the base learners, it raises the variance $\sigma^{2}$ of each of them. The number of features offered to each base learner is therefore worth tuning: too many and the decision trees agree, too few and each of them predicts poorly.

![Figure 2 of the entry randomforest](https://dictionaryofml.org/terms/fig/randomforest-fig2.svg)

*Figure 2: A random forest with three base learners, each learning a decision tree on its own random selection of the data points of the training set. For a feature vector ${\mathbf x}$, the prediction $\widehat{h}({\mathbf x})$ is obtained from an aggregation rule $\phi_{\rm agg}$ applied to the tree predictions $\widehat{h}^{(1)}({\mathbf x}), \,\ldots,\, \widehat{h}^{(3)}({\mathbf x})$*

A random forest also yields a feature importance score for each feature: the fraction of the overall splits that use that feature. For the bank, these scores name the attributes of an application that its forest tests most often. Fig. 3 computes these scores for a forest grown on four features of a day at Krems an der Donau.

![Figure 3 of the entry randomforest](https://dictionaryofml.org/terms/fig/randomforest-fig3.svg)

*Figure 3: Feature importance scores of a forest of $60$ decision trees grown on four features of a day at Krems an der Donau, predicting the maximum air temperature. Each base learner was offered randomly selected days and a randomly selected pair of the four features, and every split it made was recorded. The score of a feature is the share of the $420$ splits that test it. Data generated by pythondemos/randomforest.py*

**See also:** [ensemble](https://dictionaryofml.org/terms/ensemble.html), [bagging](https://dictionaryofml.org/terms/bagging.html), [decision tree](https://dictionaryofml.org/terms/decisiontree.html), [bootstrap](https://dictionaryofml.org/terms/bootstrap.html), [feature importance](https://dictionaryofml.org/terms/featureimportance.html), [variance](https://dictionaryofml.org/terms/variance.html), [correlation](https://dictionaryofml.org/terms/correlation.html).

## References

- Breiman (2001). Random Forests. *Machine Learning*. <https://doi.org/10.1023/A:1010933404324>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>

## Cite this entry

```bibtex
@misc{dictml_randomforest,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {random forest},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/randomforest.html}
}
```


---

# regularization

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/regularization.html) · [typeset PDF](https://dictionaryofml.org/terms/regularization.pdf) · [LaTeX](https://dictionaryofml.org/terms/regularization.tex) · [Python demo](https://dictionaryofml.org/terms/regularization.py) · [notebook](https://dictionaryofml.org/terms/regularization.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Regularization refers to modifications to a machine learning method that improve its generalization. Three routes are distinguished by which component of the method is modified: pruning the hypothesis space, adding a penalty term to the loss function, and augmenting the training set with perturbed copies of its data points. The three routes often yield the same learned hypothesis. For example, augmenting the training set of linear regression with Gaussian feature perturbations is equivalent to adding the ridge regression penalty term to the training error.

## Definition

Training a large deep net (such as a large language model) on a fixed training set via plain empirical risk minimization typically leads to overfitting: the learned hypothesis $\widehat{h}$ performs well on the training set but poorly outside it. Regularization refers to modifications to a machine learning method to ensure the learned hypothesis performs nearly as well on data points outside the training set as on those in it (Goodfellow et al., 2016, Ch. 7).

For empirical risk minimization-based methods, regularization can be done in three ways:

- Model pruning: shrink the hypothesis space $\mathcal{H}$ (the set of candidate hypotheses $h$) to a smaller $\mathcal{H}'$. For a parametric model, the shrinkage can be implemented via constraints on the model parameters (e.g., $w_{1} \in [0.4,0.6]$ on the weight of feature $x_{1}$ in linear regression).
- Loss penalization: add a penalty term to the training error of empirical risk minimization. For the squared error loss, the penalty term is an estimate of how much the risk exceeds the average loss on the training set (Hastie et al., 2009, Sect. 7.4); for other losses it controls the generalization gap without estimating it (Shalev-Shwartz and Ben-David, 2014, Cor. 13.6) (see support vector machine).
- Data augmentation: enlarge the training set $\mathcal{D}^{(\rm train)}$ with perturbed copies of its data points (e.g., adding the realizations of independent and identically distributed RVs to the feature vector of each data point).

Fig. 1 illustrates the three routes.

![Figure 1 of the entry regularization](https://dictionaryofml.org/terms/fig/regularization-fig1.svg)

*Figure 1: Three routes to regularization of an empirical risk minimization-based method, shown for linear regression. 1) Model pruning: constraining the model parameters (here $w_{1} \in [0.4,0.6]$) shrinks the hypothesis space to the hypotheses inside the blue ellipse. 2) Loss penalization: the red penalty term $\alpha \mathcal{R}\big\{ h \big\}$ is added to the blue training error near the bottom of the plot. 3) Data augmentation: the original data points (blue) are augmented with perturbed copies (red), displaced by an amount $\propto \sqrt{\alpha}$ (double arrow shown for the rightmost group)*

These routes can yield the same learned hypothesis. As an example, consider data augmentation for linear regression that adds zero-mean perturbations with covariance matrix $\sigma^{2} \mathbf{I}$ to the feature vector of each data point. Asymptotically in the number of perturbations, the resulting learned hypothesis coincides with the one obtained from ridge regression (Bishop, 1995; Goodfellow et al., 2016, Sect. 7.5). Ridge regression adds the penalty term $\sigma^{2} \left\lVert {\mathbf w} \right\rVert_{2}^{2}$ to the training error of linear regression (Hastie et al., 2009, Sect. 3.4.1). According to Lagrangian duality, adding this penalty term is equivalent to shrinking the hypothesis space to the set of hypotheses with $\left\lVert {\mathbf w} \right\rVert_{2}^{2} \leq C$ for some constant $C$ (Hastie et al., 2009, Eq. (3.42)).

The above regularization techniques are not limited to empirical risk minimization-based methods but can also be applied to other types of machine learning algorithms. For example, in online learning, the Follow-The-Leader method can be regularized by adding a regularizer to the per-round objective function of Follow-The-Leader. The resulting Follow-The-Regularized-Leader method achieves lower regret than plain Follow-The-Leader (Hazan, 2022, Ch. 5). Another instance of regularization is the use of a prior distribution over $\mathcal{H}$ in Bayesian inference. The prior distribution acts as a regularizer: the negative logarithm of the prior distribution is a penalty term, so the maximum a posteriori estimate of the model parameters of linear regression under a Gaussian prior distribution is the ridge regression solution (Bishop, 2006, Sect. 1.2.5).

**See also:** [overfitting](https://dictionaryofml.org/terms/overfitting.html), [generalization](https://dictionaryofml.org/terms/generalization.html), model pruning, penalty term, [data augmentation](https://dictionaryofml.org/terms/dataaug.html), ridge regression, least absolute shrinkage and selection operator.

## References

- Goodfellow et al. (2016). Deep Learning. *MIT Press*.
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Bishop (1995). Training with Noise is Equivalent to Tikhonov Regularization. *Neural Computation*. <https://doi.org/10.1162/neco.1995.7.1.108>
- Hazan (2022). Introduction to Online Convex Optimization. *MIT Press*. <https://sites.google.com/view/intro-oco/>
- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>

## Cite this entry

```bibtex
@incollection{dictml_regularization,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {regularization},
  booktitle = {Dictionary of Applied Machine Learning},
  publisher = {Springer Nature Singapore},
  year = {2026},
  doi = {10.1007/978-981-92-2012-0_8-1},
  isbn = {978-981-92-2012-0},
  url = {https://doi.org/10.1007/978-981-92-2012-0_8-1}
}
```


---

# reinforcement learning (RL)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/reinforcementlearning.html) · [typeset PDF](https://dictionaryofml.org/terms/reinforcementlearning.pdf) · [LaTeX](https://dictionaryofml.org/terms/reinforcementlearning.tex) · [Python demo](https://dictionaryofml.org/terms/reinforcementlearning.py) · [notebook](https://dictionaryofml.org/terms/reinforcementlearning.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Reinforcement learning (RL) is the task of learning to act optimally within some environment. The actions are taken by an agent based on observations about the environment. The standard mathematical model of the interaction between agent and environment is a Markov decision process. An MDP includes a transition function, the conditional probability distribution of the next state given the current state and action. Another component of an MDP is a reward function that maps each pair of state and action to a reward value, which can be interpreted as a negative loss value. RL differs from supervised learning in the construction of the loss function: at each time step, the reward is observed only for the action taken, and it depends on previous actions, which forces a trade-off between exploring new actions and exploiting the ones found best.

## Definition

Consider a robot on the $4 \times 4$ gridworld of Fig. 1: a grid of cells, the cell in row $r$ and column $c$ written $\langle r, c \rangle$, with a wall at $\langle 2, 2 \rangle$ and a goal at $\langle 3, 3 \rangle$. At each time step, the robot moves up, down, left or right. A move into the wall or the boundary leaves it in place. Every move yields a reward of $-1$, while the move that enters the goal yields a reward of $+1$ and results in a stand still. RL is the task of learning, from such interaction, to choose actions that eventually reach the goal. The learner is not told which action is correct but must discover the actions that yield the most reward by trying them, and an action affects not only the immediate reward but also the next state and, through it, all later rewards. These two properties, trial-and-error search and delayed reward, distinguish RL from other forms of learning (Sutton and Barto, 2018, Sect. 1.1).

The standard mathematical model of an RL application is a Markov decision process (Sutton and Barto, 2018, Sect. 1.1): a state space $\mathcal{S}$, an action space $\mathcal{A}$, the conditional probability distribution of the next state and reward given the current state and action, and a discount factor $\gamma \in [0, 1]$. The conditional probability distribution of the next state alone is the transition function. An agent interacts with its environment in time steps $t = 1, 2, \ldots$: it observes the state $s_{t}$, chooses an action $a_{t}$, and receives the reward $r_{t+1}$ and the next state $s_{t+1}$ (Sutton and Barto, 2018, Sect. 3.1). The actions are chosen by a policy $\pi$, which for a stochastic policy is a conditional probability distribution $\mathbb{P}^{(a \mid s)}$ over the actions given the state. The quantity to be maximized is the return, the discounted sum of the rewards to come, $G_{t} = \sum_{k=0}^{\infty} \gamma^{k} r_{t+k+1}$, where the discount factor $\gamma$ weighs later rewards less (Sutton and Barto, 2018, Sect. 3.3).

The state-value function of a policy,

$$
v_{\pi}(s) :=
\mathbb{E}_{\pi} \big\{ G_{t} \mid s_{t} = s \big\} \text{,}
$$

is the expected return from a state when the actions are chosen by $\pi$ (Sutton and Barto, 2018, Sect. 3.5). An optimal policy $\pi_{*}$ is one whose state-value function is at least that of every other policy at every state (Sutton and Barto, 2018, Sect. 3.6). In the gridworld with $\gamma = 1$, a shortest route of $d$ moves earns $-1$ for each of its first $d - 1$ moves and $+1$ for the last, so $v_{\pi_{*}}(s) = -(d-1) + 1
= 2 - d$.

![Figure 1 of the entry reinforcementlearning](https://dictionaryofml.org/terms/fig/reinforcementlearning-fig1.svg)

*Figure 1: The $4 \times 4$ gridworld of the opening example: a wall at $\langle 2, 2 \rangle$, the goal at $\langle 3, 3 \rangle$, reward $-1$ per move and $+1$ for the move that enters the goal. Each cell shows the optimal state-value function $v_{\pi_{*}}(s)$, which equals $2$ minus the number of moves to the goal, and the arrow of the optimal policy $\pi_{*}$. Both are learned from the rewards of the moves tried, without being told how a move changes the state. Data generated by pythondemos/reinforcementlearning.py*

As a special case of machine learning, RL also trains a model on data: a policy is the hypothesis of RL. The two properties named above set it apart from supervised learning. The first, trial-and-error search, is the construction of the loss function. A supervised learning method holds the label of each data point and can evaluate the loss of every candidate hypothesis and its predictions. An RL method can evaluate the loss, the negative reward, only for the action it took, since the rewards of the actions not taken are not observed. The second, delayed reward, is that the loss at time step $t$ depends on the previous actions. RL is not unsupervised learning either: it maximizes a reward signal instead of finding structure in unlabeled data points (Sutton and Barto, 2018, Sect. 1.1).

Within RL itself, methods differ in the amount of information about the environment required upfront, from a known Markov decision process to nothing beyond the observed rewards (Sutton and Barto, 2018, Sect. 1.3). If the Markov decision process is known, methods such as value iteration and policy iteration compute an optimal policy before the agent executes it (Sutton and Barto, 2018, Ch. 4). A policy is greedy with respect to a value function $v$ if it maximizes, at every state $s \in \mathcal{S}$, the expected immediate reward plus the discounted value of the next state. It therefore chooses an action in $\mathop{\mathrm{arg\,max}}_{a \in \mathcal{A}} \mathbb{E} \big\{ r_{t+1}
+ \gamma v(s_{t+1}) \mid
s_{t} = s, a_{t} = a \big\}$ (Sutton and Barto, 2018, Sect. 4.2). Value iteration and policy iteration use two facts. The state-value function of a policy is the fixed point of its Bellman operator, which iterative policy evaluation reaches from any starting value function (Sutton and Barto, 2018, Sect. 4.1). And the policy that is greedy with respect to $v_{\pi}$ is at least as good as $\pi$ (Sutton and Barto, 2018, Sect. 4.2).

When the Markov decision process is not known, an RL method has to learn from the interaction itself, and every reward it observes is that of an action it has taken. This forces a trade-off between exploration, trying actions whose reward is unknown, and exploitation, choosing the actions found best so far (Sutton and Barto, 2018, Sect. 1.1). Q-learning learns the action-value function $q_{\pi_{*}}(s, a)$, the expected return of taking $a$ in $s$ and following $\pi_{*}$ afterwards, from the rewards of the actions it tries, updating after each time step (Sutton and Barto, 2018, Sect. 6.5). Other methods learn the policy directly (Sutton and Barto, 2018, Ch. 13), or estimate a model of the environment and plan with it (Sutton and Barto, 2018, Ch. 8).

Two special cases of the Markov decision process remove the delayed reward and leave only the trade-off between exploration and exploitation (Sutton and Barto, 2018, Sect. 2.1). The first has a single state: the transition function yields that state whatever the action, so the return is a sum of rewards that each depend on the current action alone. This is the multiarmed bandit (Sutton and Barto, 2018, Ch. 2). An ad server that shows one of ten banners and observes the clicks of the banner shown, but not of the nine it withheld, faces this case (Bubeck and Cesa-Bianchi, 2012). The upper confidence bound method handles it by adding to the sample mean of each action a confidence term that shrinks with the number of times the action was chosen (Auer et al., 2002, Thm. 1; Bubeck and Cesa-Bianchi, 2012).

The second case keeps several states but draws $s_{t}$ from a fixed probability distribution that does not depend on the actions taken. The action at time $t$ then changes the reward $r_{t+1}$ and nothing else, and the state is called the context (Sutton and Barto, 2018, Sect. 2.9). This is the contextual bandit. The learning task is to learn a hypothesis that maps a context to an action, which has the form of a supervised learning problem, except that the loss is still observed only for the action taken. News recommendation is an instance: the context is the visitor profile, the action is the article placed on the front page, and the reward is the click (Li et al., 2010).

RL is often a form of online learning: the agent learns while it acts, and the data points it learns from arrive one per time step and depend on its current policy (Sutton and Barto, 2018, Sect. 1.1). Offline RL instead learns a policy from a fixed dataset of recorded interactions, without acting in the environment during training (Levine et al., 2020). The exploration is then fixed by whoever collected the dataset.

**See also:** Markov decision process, [policy](https://dictionaryofml.org/terms/policy.html), [reward](https://dictionaryofml.org/terms/reward.html), return, value function, state-value function, gridworld, agent, environment, Bellman operator, Q-learning, policy iteration, value iteration, multiarmed bandit, contextual bandit, upper confidence bound, [online learning](https://dictionaryofml.org/terms/onlinelearning.html), [supervised learning](https://dictionaryofml.org/terms/supervisedlearning.html), [unsupervised learning](https://dictionaryofml.org/terms/unsupervisedlearning.html).

## References

- Sutton and Barto (2018). Reinforcement Learning: An Introduction. *MIT Press*.
- Bubeck and Cesa-Bianchi (2012). Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems. *Found. Trends Mach. Learn.*. <https://doi.org/10.1561/2200000024>
- Auer et al. (2002). Finite-time Analysis of the Multiarmed Bandit Problem. *Machine Learning*.
- Li et al. (2010). A Contextual-Bandit Approach to Personalized News Article Recommendation. *Proc. International World Wide Web Conference*.
- Levine et al. (2020). Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. <https://arxiv.org/abs/2005.01643>

## Cite this entry

```bibtex
@misc{dictml_reinforcementlearning,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {reinforcement learning (RL)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/reinforcementlearning.html}
}
```


---

# reward

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-04. [HTML](https://dictionaryofml.org/terms/reward.html) · [typeset PDF](https://dictionaryofml.org/terms/reward.pdf) · [LaTeX](https://dictionaryofml.org/terms/reward.tex) · [Python demo](https://dictionaryofml.org/terms/reward.py) · [notebook](https://dictionaryofml.org/terms/reward.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A reward is an observed quantity that allows a machine learning method to estimate the loss incurred by the prediction of a hypothesis. One example of such a quantity is the label of a data point itself. Many machine learning applications, however, do not allow the label to be observed, but only a noisy estimate of it; some allow only the prediction error of one specific prediction to be observed. Reinforcement learning methods learn a sequence of hypothesis maps from the rewards obtained by their predictions. For example, a self-driving vehicle predicts, at each time instant, a steering direction, and observes as a reward whether that direction keeps the vehicle on its lane, with a low reward for a direction that carries it off the road.

## Definition

A self-driving car has to choose a steering direction at each time instant, and the optimal direction is never observed; what is observed is whether the chosen direction keeps the car on its lane (Fig. 1). The ultimate goal of machine learning is to learn a hypothesis that predicts the label of a data point well from its features, and choosing among different hypothesis maps requires a measure of the prediction error that a hypothesis incurs. Such a measure is the loss, which can be defined, for example, by comparing the prediction $h({\mathbf x})$ with the actual label value $y$. For the car, a data point is the current time instant and its label is the optimal steering direction, which cannot be accessed at all.

![Figure 1 of the entry reward](https://dictionaryofml.org/terms/fig/reward-fig1.svg)

*Figure 1: A self-driving car at three time instants, between the two contours of the road. At each instant the hypothesis predicts a steering direction (black arrows); at time instant $t$ that prediction is $h^{(t)}\big({\mathbf x}^{(t)}\big)$. The red arrow is the label $y^{(t+2)}$, the optimal steering direction, which follows the lane. That label is rarely accessible: obtaining it would require a driving instructor to analyze every possible driving scenario. What is observed instead is a reward: at $t$ and $t+1$ the predicted direction keeps the car on its lane, while at $t+2$ it carries the car over the outer contour and earns a low reward*

A reward is an observed quantity from which a machine learning method estimates the loss incurred by the prediction of a hypothesis. One example of such a quantity is the label of a data point itself. Many machine learning applications, however, do not allow the label to be observed, but only a noisy estimate of it. Some allow only the prediction error of one specific prediction to be observed. The self-driving car of Fig. 1 is of the last kind: the reward is whether the steering direction $h({\mathbf x})$ predicted at a time instant keeps the car on its lane, and a low reward is assigned to a direction that carries the car over a contour of the road.

A reward replaces the label of supervised learning: it does not reveal the correct prediction, only how good the chosen prediction, the action, was, and it may arrive delayed, several time steps after the action that caused it, or corrupted by noise (Sutton and Barto, 2018, Sect. 1.1). Reinforcement learning methods learn a sequence of hypothesis maps from the rewards obtained by their predictions: the negative reward is the loss of the action taken, and the actions chosen are those that maximize the cumulative reward, the return (Sutton and Barto, 2018, Sect. 3.2). In the self-driving car of Fig. 1, the state $s_{t}$ is the position and orientation of the car on the road at time instant $t$, and the action $a_{t}$ is the steering direction predicted there. Fig. 2 shows how states, actions and rewards follow one another over two time steps.

![Figure 2 of the entry reward](https://dictionaryofml.org/terms/fig/reward-fig2.svg)

*Figure 2: The working principle of reinforcement learning. At each time step $t$ the state $s_{t}$ is observed, an action $a_{t}$ is chosen, and a reward $r_{t+1}$ is received (solid arrows). The reward scores the chosen action without revealing which action would have been best, and an action can also affect a later reward (dashed arrow), which is why the return, the cumulative reward, rather than the next reward alone is maximized*

**See also:** [loss](https://dictionaryofml.org/terms/loss.html), multiarmed bandit, [reinforcement learning](https://dictionaryofml.org/terms/reinforcementlearning.html), return, action, [label](https://dictionaryofml.org/terms/label.html).

## References

- Sutton and Barto (2018). Reinforcement Learning: An Introduction. *MIT Press*.

## Cite this entry

```bibtex
@misc{dictml_reward,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {reward},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/reward.html}
}
```


---

# robust principal component analysis (robust PCA)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/rpca.html) · [typeset PDF](https://dictionaryofml.org/terms/rpca.pdf) · [LaTeX](https://dictionaryofml.org/terms/rpca.tex) · [Python demo](https://dictionaryofml.org/terms/rpca.py) · [notebook](https://dictionaryofml.org/terms/rpca.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Robust principal component analysis (robust PCA) decomposes a feature matrix into a low-rank matrix, whose rows lie in a subspace of small dimension, and a sparse matrix that carries a few corrupted feature values. Unlike principal component analysis, it does not compute the low-rank matrix from the recorded values, so a large fault lands in the sparse matrix instead of turning the subspace. One approach states robust PCA as a convex optimization problem, referred to as principal component pursuit. Convex optimization methods solve it, among them the alternating direction method of multipliers. The resulting method delivers estimates for the principal directions that span the subspace, and for the corrupted entries in the features of each data point.

## Definition

The weather station Krems an der Donau records the air temperature every hour, so ten days of 2024 give ten rows of $24$ values (Fig. 1). Stacking the $24$ values of one day gives the feature vector ${\mathbf x} \in \mathbb{R}^{d}$, with $d = 24$, of the corresponding data point. Stacking the $m = 10$ feature vectors row-wise gives the $m \times d$ feature matrix ${\mathbf X}$. Each feature vector is centered by subtracting the average of the $m$ feature vectors, so that this average is the zero vector.

Principal component analysis uses the feature matrix to learn a given number $d'$ of orthonormal vectors $\mathbf{u}^{(1)}, \ldots, \mathbf{u}^{(d')}$, the principal directions of ${\mathbf X}$. They span a $d'$-dimensional subspace $\mathcal{U}$ of $\mathbb{R}^{d}$ such that each feature vector ${\mathbf x}^{(r)}$, for $r = 1, \ldots, m$, can be well approximated by its projection

$$
\mathbf{P} {\mathbf x}^{(r)} =
\sum_{j=1}^{d'}
\langle \mathbf{u}^{(j)}, {\mathbf x}^{(r)} \rangle
\mathbf{u}^{(j)} \text{.}
$$

Here, $\mathbf{P} \in \mathbb{R}^{d \times d}$ denotes the orthogonal projection matrix on the subspace $\mathcal{U}$. It is given explicitly by $\mathbf{P}=\sum_{j=1}^{d'}
\mathbf{u}^{(j)} \big( \mathbf{u}^{(j)} \big)^{\top}$ and provides a fully equivalent representation of the subspace $\mathcal{U}$ spanned by the principal directions.

The projection matrix $\mathbf{P}$ delivered by principal component analysis decomposes each feature vector as ${\mathbf x}^{(r)} = \mathbf{P} {\mathbf x}^{(r)} + {\mathbf e}^{(r)}$, with the approximation error ${\mathbf e}^{(r)}$. These decompositions (one for each $r=1,\ldots,m$) can be expressed more compactly in terms of the feature matrix as

\begin{equation}
\tag{1}
{\mathbf X} = {\mathbf L} + \mathbf{E} \text{, where } {\mathbf L} = {\mathbf X} \mathbf{P}
\text{ and } \mathbf{E} = {\mathbf X} - {\mathbf L} \text{.}
\end{equation}

The rank of ${\mathbf L}$ is at most $d'$, since every row of ${\mathbf L}$ lies in the subspace $\mathcal{U}$. The error matrix $\mathbf{E}$ carries the ${\mathbf e}^{(r)}$ as its rows, so $\left\lVert \mathbf{E} \right\rVert_{F}^{2} = \sum_{r=1}^{m}
\left\lVert {\mathbf e}^{(r)} \right\rVert_{2}^{2}$. Principal component analysis chooses $\mathbf{P}$ to make this quantity as small as possible. In this reading, principal component analysis splits the feature matrix into a low-rank part ${\mathbf L}$ and a small remainder $\mathbf{E}$.

The decomposition Eq. (1) suggests using a representation of the component $\mathbf{P} {\mathbf x}^{(r)}$ as new features. One such representation is the vector ${\mathbf z}^{(r)} = \big(
\langle \mathbf{u}^{(1)}, {\mathbf x}^{(r)} \rangle, \ldots,
\langle \mathbf{u}^{(d')}, {\mathbf x}^{(r)} \rangle
\big)^{\top}$ whose entries are the inner products between ${\mathbf x}^{(r)}$ and the principal directions of the feature matrix ${\mathbf X}$. For $d' = 2$ the new features $z_{1}, z_{2}$ can be used as coordinates in a scatterplot, as panel (a) of Fig. 2 shows.

Sometimes a feature value is faulty, for example because a temperature sensor fails. The red square in Fig. 1 marks such a failure: the noon reading of day five is increased by $22$ degrees, which gives the perturbed feature vector $\widetilde{{\mathbf x}}^{(5)}$ and, in turn, the perturbed feature matrix $\widetilde{{\mathbf X}}$. Panel (b) of Fig. 2 shows the scatterplot that principal component analysis delivers from $\widetilde{{\mathbf X}}$. The new features $z^{(5)}_{1}, z^{(5)}_{2}$ obtained for $\widetilde{{\mathbf x}}^{(5)}$ are far from those of the other days, and the second learned principal direction turns toward the axis of the perturbed feature. Thus, an isolated but large fault can substantially distort the coordinates of every data point.

The decomposition Eq. (1) explains why. Applied to $\widetilde{{\mathbf X}}$, it has the low-rank part ${\mathbf L} = \widetilde{{\mathbf X}} \mathbf{P}$, which is computed from the recorded values, and a remainder $\mathbf{E}$ that principal component analysis keeps small in the Frobenius norm. The $22$ degrees can therefore not be left in $\mathbf{E}$. Most of them have to appear in ${\mathbf L}$, and the only way a matrix of rank at most $d'$ can carry a large value at a single entry is for the subspace $\mathcal{U}$ to turn toward the faulty feature.

![Figure 1 of the entry rpca](https://dictionaryofml.org/terms/fig/rpca-fig1.svg)

*Figure 1: Hourly temperature recordings for ten days of 2024 at Krems an der Donau, drawn as small black dots. Each day is a data point with a $24$-dimensional feature vector ${\mathbf x}$. The red circle marks the correct noon reading of day five, and the large red square the perturbation that replaces it, $22$ degrees higher*

Robust principal component analysis drops the constraint ${\mathbf L} = {\mathbf X} \mathbf{P}$ and adds a third summand for the faulty entries,

\begin{equation}
\tag{2}
{\mathbf X} = {\mathbf L} + \mathbf{S} + \mathbf{E} \text{.}
\end{equation}

The low-rank matrix ${\mathbf L}$ is now an unknown in its own right, constrained only by $\mathop{\mathrm{rank}}({\mathbf L}) \leq d'$. The sparse matrix $\mathbf{S}$ carries the faulty feature values and is zero in every other entry. The remainder $\mathbf{E}$ stays small, as in the principal component analysis decomposition Eq. (1). Row $r$ of the decomposition Eq. (2) splits the feature vector ${\mathbf x}^{(r)}$ into these three parts.

The subspace and the principal directions are computed from ${\mathbf L}$ afterwards. The rows of ${\mathbf L}$ span a subspace of dimension at most $d'$, and the principal directions $\mathbf{u}^{(1)}, \ldots, \mathbf{u}^{(d')}$ are the leading right singular vectors of ${\mathbf L}$, delivered by its singular value decomposition. The projection matrix $\mathbf{P}$ follows from them, and ${\mathbf L} = {\mathbf L} \mathbf{P}$ holds because every row of ${\mathbf L}$ already lies in the subspace that $\mathbf{P}$ projects on. Robust principal component analysis therefore delivers the same objects as principal component analysis, but from a matrix ${\mathbf L}$ that is free of the faulty feature values, since $\mathbf{S}$ carries them.

The goal of making $\mathbf{E}$ small while $\mathbf{S}$ carries only a few non-zero entries can be stated as an optimization problem

\begin{equation}
\tag{3}
\min_{{\mathbf L}, \mathbf{S} \in \mathbb{R}^{m \times d}}
\left\lVert {\mathbf X} - {\mathbf L} - \mathbf{S} \right\rVert_{F} \quad
\text{ subject to } \mathop{\mathrm{rank}}({\mathbf L}) \leq d', \;
\left\lVert \mathbf{S} \right\rVert_{0} \leq s \text{.}
\end{equation}

The optimization variables are the two matrices ${\mathbf L}$ and $\mathbf{S}$, and the objective function is the Frobenius norm of the remainder $\mathbf{E} = {\mathbf X} - {\mathbf L} - \mathbf{S}$. The pseudo-norm $\left\lVert \mathbf{S} \right\rVert_{0}$ denotes the number of non-zero entries of $\mathbf{S}$, and $s$ is a given bound on the number of faulty feature values.

The optimization problem Eq. (3) is non-convex, since neither the rank nor the number of non-zero entries is a convex function. Principal component pursuit replaces it with the convex optimization problem

\begin{equation}
\tag{4}
\min_{{\mathbf L}, \mathbf{S} \in \mathbb{R}^{m \times d}}
\left\lVert {\mathbf L} \right\rVert_{*} + \alpha \left\lVert \mathbf{S} \right\rVert_{1} \quad
\text{ subject to } {\mathbf X} = {\mathbf L} + \mathbf{S} \text{,}
\end{equation}

with a regularization parameter $\alpha > 0$ (Candès et al., 2011). Here, $\left\lVert {\mathbf L} \right\rVert_{*}$ is the sum of the singular values of ${\mathbf L}$, and $\left\lVert \mathbf{S} \right\rVert_{1}$ the sum of the absolute values of the entries of $\mathbf{S}$. These two norms are convex, as every norm is (Boyd and Vandenberghe, 2004, Sects. 3.1.5 and A.1.6), and take the place of the rank of ${\mathbf L}$ and of the number of non-zero entries of $\mathbf{S}$. A large regularization parameter $\alpha$ makes every non-zero entry of $\mathbf{S}$ expensive, so the solution keeps $\mathbf{S}$ sparser and leaves more of ${\mathbf X}$ to ${\mathbf L}$. The rank of ${\mathbf L}$, and with it the dimension $d'$ of the subspace, then comes out larger, while a small $\alpha$ has the opposite effect.

Robust principal component analysis methods differ in the optimization method used to solve principal component pursuit Eq. (4). Algorithm 1 summarizes one such method, obtained by applying the alternating direction method of multipliers. Panel (c) of Fig. 2 shows the scatterplot that principal component analysis delivers from the low-rank matrix ${\mathbf L}$ that Algorithm 1 recovers from the perturbed feature matrix $\widetilde{{\mathbf X}}$. The ten days are back at the coordinates of panel (a). The non-zero entries of $\mathbf{S}$ point at the faulty readings themselves, which is why robust principal component analysis is also used for anomaly detection.

![Figure 2 of the entry rpca](https://dictionaryofml.org/terms/fig/rpca-fig2.svg)

*Figure 2: Coordinates of the same ten days given by the two features learned by (a) principal component analysis on the clean readings, (b) principal component analysis after one noon reading of day five is increased by $22$ degrees, and (c) principal component analysis on the low-rank matrix ${\mathbf L}$ that robust principal component analysis recovers from the corrupted readings. A cross marks day five in every panel, a filled circle each of the other nine days. All three panels share one set of axis limits, so the point clouds can be compared directly*

Algorithm 1 is a fixed-point iteration

$$
\big({\mathbf L}^{(t+1)}, \mathbf{S}^{(t+1)},
{\mathbf W}^{(t+1)}\big) = T\big({\mathbf L}^{(t)}, \mathbf{S}^{(t)},
{\mathbf W}^{(t)}\big) \text{,}
$$

whose operator $T$ chains two proximal operators with the update of the matrix ${\mathbf W}$. Shrinking the singular values of ${\mathbf B}$ by $1/\rho$ is the proximal operator of the nuclear norm (Candès et al., 2011, Eq. (5.3)), which is why that step needs the singular value decomposition. Shrinking every entry of ${\mathbf C}$ by $\alpha/\rho$ is the proximal operator of $\left\lVert \cdot \right\rVert_{1}$ (Candès et al., 2011, Eq. (5.2)), the one that solvers for the least absolute shrinkage and selection operator also use and the one step where $\alpha$ enters the iteration. At a fixed point of $T$ the three updates leave the iterate unchanged. This happens precisely when ${\mathbf X} = {\mathbf L} + \mathbf{S}$ holds, $({\mathbf L}, \mathbf{S})$ solves principal component pursuit Eq. (4), and $\rho {\mathbf W}$ is a Lagrange multiplier of the constraint. The optimization problem Eq. (4) is convex, so the iteration converges for every $\rho > 0$: the residual ${\mathbf X} - {\mathbf L} - \mathbf{S}$ tends to zero and the objective function to its minimum (Boyd et al., 2011, Sect. 3.2). For the averages of the iterates over the first $T$ iterations, this convergence holds with an explicit bound (Beck, 2017, Thm. 15.4).

![Algorithm 1 of the entry rpca](https://dictionaryofml.org/terms/fig/rpca-alg1.svg)

*Algorithm 1: Principal component pursuit by alternating direction method of multipliers (Candès et al., 2011, Sect. 5, Algorithm 1)*

**See also:** [principal component analysis](https://dictionaryofml.org/terms/pca.html), sparse principal component analysis, probabilistic principal component analysis, [singular value decomposition](https://dictionaryofml.org/terms/svd.html), rank, alternating direction method of multipliers, proximal operator, fixed-point iteration, least absolute shrinkage and selection operator, [regularization](https://dictionaryofml.org/terms/regularization.html), outlier, anomaly detection, robustness.

## References

- Candès et al. (2011). Robust Principal Component Analysis?. *Journal of the ACM*. <https://doi.org/10.1145/1970392.1970395>
- Boyd and Vandenberghe (2004). Convex Optimization. *Cambridge Univ. Press*. <https://doi.org/10.1017/CBO9780511804441>
- Boyd et al. (2011). Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers. *Found. Trends Mach. Learn.*. <https://doi.org/10.1561/2200000016>
- Beck (2017). First-Order Methods in Optimization. *SIAM-Society for Industrial and Applied Mathematics*.

## Cite this entry

```bibtex
@misc{dictml_rpca,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {robust principal component analysis (robust PCA)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/rpca.html}
}
```


---

# random variable (RV)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/rv.html) · [typeset PDF](https://dictionaryofml.org/terms/rv.pdf) · [LaTeX](https://dictionaryofml.org/terms/rv.tex) · [Python demo](https://dictionaryofml.org/terms/rv.py) · [notebook](https://dictionaryofml.org/terms/rv.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A random variable (RV) is a measurable function that maps each outcome of a random experiment to an element of a measurable space. Important types of RVs include binary RVs, discrete RVs, real-valued RVs, random vectors, and random matrices. The analysis of machine learning methods often relies on probabilistic models that represent data sources as RVs. The most widely used probabilistic model assumes that data points are independent and identically distributed RVs with a common probability distribution, which is unknown in general.

## Definition

The weather on a given day in the Austrian city Krems an der Donau is the outcome of a random experiment (Hyndman and Athanasopoulos, 2021). Each possible outcome $\omega$ is characterized by a maximum daytime temperature, the temperature at 13:00, and recording this temperature assigns a real number $x(\omega)$ to the outcome. This assignment is an RV $x$. The temperatures recorded on consecutive days are realizations of this RV (see Fig. 1).

![Figure 1 of the entry rv](https://dictionaryofml.org/terms/fig/rv-fig1.svg)

*Figure 1: Each day's maximum daytime temperature, read at 13:00, is a realization of the real-valued RV $x$; the stem plot shows one realization per day over 30 consecutive days. Data generated by pythondemos/rv.py*

More generally, an RV maps the outcomes of a random experiment to elements of a measurable space (Billingsley, 1986; Gray, 2009). Mathematically, an RV is a measurable function $x: \Omega \rightarrow \mathcal{X}$ whose domain is the sample space $\Omega$ of a probability space and whose co-domain is a measurable space $\mathcal{X}$. Measurability refers to the σ-algebras of the two spaces: the preimage $x^{-1}(B)$ of every measurable set $B \subseteq \mathcal{X}$ must belong to the σ-algebra of the probability space, so that the probability of $x$ taking a value in $B$ is defined. The value $x(\omega) \in \mathcal{X}$ that $x$ assigns to an outcome $\omega \in \Omega$ is the image of $\omega$ under $x$ (see Fig. 2).

![Figure 2 of the entry rv](https://dictionaryofml.org/terms/fig/rv-fig2.svg)

*Figure 2: An RV $x$ sends each outcome $\omega$ of a random experiment to its image $x(\omega)$ in a measurable space $\mathcal{X}$. The black arrow follows one outcome; the gray arrows show the remaining outcomes, two of which share the same image. An RV can therefore map a large sample space into a small measurable space*

Different types of RVs include

- binary RVs, which map each outcome to an element of a binary set (e.g., $\{-1,1\}$ or $\{\text{cat}, \text{no cat}\}$);
- discrete RVs, which take on values in a countable set (which can be finite or countably infinite; e.g., the number of heads in ten coin flips, with values in $\{0,1,\ldots,10\}$);
- real-valued RVs, which take on values in the real numbers $\mathbb{R}$ (e.g., the maximum daytime temperature in Krems);
- random vectors, which map outcomes to the Euclidean space $\mathbb{R}^{d}$ (e.g., the red-green-blue (RGB) values of a pixel, which form a vector in $\mathbb{R}^{3}$);
- random matrices, which map outcomes to a space of matrices $\mathbb{R}^{m \times d}$ (e.g., the data matrix of a dataset).

The sample space and the RVs on it are a modeling choice, not fixed by the data. The daily Krems temperatures admit two standard constructions (Fig. 3). In the first, each day $r$ has its own probability space with sample space $\Omega^{(r)}$, and the temperature of that day is an RV $x^{(r)} : \Omega^{(r)} \rightarrow \mathbb{R}$; the recorded value is one realization $x^{(r)}(\omega^{(r)})$. In the second, one probability space with sample space $\Omega$ represents all days at once: a single outcome $\omega \in \Omega$ fixes the temperatures of all days, and each day is an RV $x^{(r)} : \Omega \rightarrow \mathbb{R}$ on this common space. The independent and identically distributed assumption is a statement about the second construction: the $x^{(r)}$ share one probability distribution and are independent, a property defined for RVs on one common probability space (Billingsley, 1986, Sect. 20). The common-space construction is the one used for learning from data, because a single probability distribution then governs an entire dataset.

![Figure 3 of the entry rv](https://dictionaryofml.org/terms/fig/rv-fig3.svg)

*Figure 3: Two constructions for the daily Krems an der Donau temperatures $t_{r}$. (a) Each day $r$ has its own probability space with sample space $\Omega^{(r)}$ and RV $x^{(r)}$, with a separate outcome per day. (b) One probability space with sample space $\Omega$ represents all days at once, so a single outcome $\omega$ fixes every day's temperature $t_{r} = x^{(r)}(\omega)$; only this construction supports modeling the days as independent and identically distributed RVs*

In machine learning, RVs represent data sources, so that the behavior of learning methods can be analyzed. For example, the generation of a training set or validation set is represented by independent and identically distributed RVs sharing one underlying probability distribution in the common-space construction. This probability distribution is either postulated or estimated from actual datasets, for example via maximum likelihood. Ideally, a machine learning method should work well on datasets that arise as realizations of high probability. This is the generalization goal that empirical risk minimization pursues, minimizing the average loss on the training set as a proxy for the expected loss of the learned hypothesis on a new data point drawn from the common probability distribution.

**See also:** function, [random experiment](https://dictionaryofml.org/terms/randomexperiment.html), sample space, probability space, vector, Euclidean space, probability, measurable, realization, probabilistic model, random vector, random matrix, sub-Gaussian random variable.

## References

- Hyndman and Athanasopoulos (2021). Forecasting: Principles and Practice. *OTexts*. <https://otexts.com/fpp3/>
- Billingsley (1986). Probability and Measure. *Wiley*.
- Gray (2009). Probability, Random Processes, and Ergodic Properties. *Springer Science+Business Media*. <https://doi.org/10.1007/978-1-4419-1090-5>

## Cite this entry

```bibtex
@misc{dictml_rv,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {random variable (RV)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/rv.html}
}
```


---

# sample size

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/samplesize.html) · [typeset PDF](https://dictionaryofml.org/terms/samplesize.pdf) · [LaTeX](https://dictionaryofml.org/terms/samplesize.tex) · [Python demo](https://dictionaryofml.org/terms/samplesize.py) · [notebook](https://dictionaryofml.org/terms/samplesize.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

The sample size is the number $m$ of data points in a sample. It is the basic measure of how much a sample carries, but not a universal one: listing a data point twice raises the count and adds nothing, correlated data points make the count overstate what a sample carries, and it says nothing about how well the data points cover the feature space. Many machine learning methods model the data points as realizations of independent and identically distributed RVs and judge a hypothesis by its expected loss, the risk; since the probability distribution is unknown, the risk is estimated by the average loss on the sample, for training by empirical risk minimization and for validation. How close that average is to the risk depends on the sample size. A concentration inequality such as Hoeffding's inequality bounds the deviation of the average from the risk and turns a required accuracy $\epsilon$ and confidence $1 - \delta$ into a sample size of order $\ln(1/\delta) / \epsilon^{2}$ for validation; for training, the sample size must also be compared with the size of the model. For data points recorded in sequence, autocorrelation lowers the effective sample size. The same reasoning gives the sample size for a statistical test, such as whether two features are correlated: the sample size needed to detect a correlation grows as the inverse square of that correlation.

## Definition

A frost warning for Krems an der Donau predicts, from the minimum and the maximum temperature of a day, whether the next day brings frost. A hypothesis $h$ for this task is judged by its expected loss, the risk $\bar{L} \big( h \big) = \mathbb{E} \{ L\left({\mathbf z},h \right) \}$, where the expectation is over a data point ${\mathbf z}$ drawn from the probability distribution $\mathbb{P}^{({\mathbf z})}$ of the days. Many machine learning methods use the independent and identically distributed assumption: the data points are realizations of independent and identically distributed RVs ${\mathbf z}^{(1)}, \ldots, {\mathbf z}^{(m)} \sim
\mathbb{P}^{({\mathbf z})}$. In practice, $\mathbb{P}^{({\mathbf z})}$ is unknown, and a method observes only these $m$ data points, a sample. Their number $m$ is the sample size.

With the probability distribution unknown, the risk is estimated by the average loss on the sample,

\begin{equation}
\tag{1}
\widehat{L}\big(h|\mathcal{D}\big) := \frac{1}{m}
\sum_{r=1}^{m}
L\left({\mathbf z}^{(r)},h \right) \text{,}
\end{equation}

which is the empirical risk. Empirical risk minimization minimizes this average over the hypotheses of a model on a training set, and validation compares the learned hypotheses by the same average on a validation set. Both uses require the average to be close to the risk, and how close it is depends on the sample size: by the law of large numbers, the average Eq. (1) converges to $\bar{L} \big( h \big)$ as $m$ grows (Papoulis and Pillai, 2002, Sect. 8-4; Shalev-Shwartz and Ben-David, 2014, Sect. 4.2). Fig. 1 shows this for the frost warning: the sample average of the 0/1 loss of one fixed hypothesis, computed on samples of $m = 10$ to $1585$ days, scatters around the risk $0.118$ with a spread that falls like $1/\sqrt{m}$. The spread is the standard deviation of the average over repeated samples, and Hoeffding's inequality, stated below, turns it into a deviation that holds with a prescribed probability.

![Figure 1 of the entry samplesize](https://dictionaryofml.org/terms/fig/samplesize-fig1.svg)

*Figure 1: The sample average of the 0/1 loss of a fixed frost-warning hypothesis on $m$ days of 2024 at Krems an der Donau, drawn uniformly with replacement, against the sample size. The solid curve is the standard deviation (spread) of the average over $400$ draws, the dotted curve $1/(2\sqrt{m})$, the largest standard deviation a loss in $[0,1]$ allows; the dashed curve is the deviation $\epsilon$ that Hoeffding's inequality Eq. (2) guarantees with probability $0.95$, and the vertical line marks the sample size $738$ at which that guarantee reaches $\epsilon = 0.05$. Data generated by pythondemos/samplesize.py*

The sample must be large enough, and one approach to deciding how large uses a concentration inequality, which bounds the probability that the average deviates from the risk by more than a given amount. For a loss function with values in $[0, 1]$, such as the 0/1 loss, Hoeffding's inequality gives, for every $\epsilon > 0$,

\begin{equation}
\tag{2}
\mathbb{P}\left( \big| \widehat{L}\big(h|\mathcal{D}\big) - \bar{L} \big( h \big) \big| > \epsilon \right)
\leq 2 \exp\big( -2 m \epsilon^{2} \big) \text{,}
\end{equation}

see Hoeffding (1963) and Shalev-Shwartz and Ben-David (2014, Lemma 4.5). The bound holds for a hypothesis chosen before the sample is drawn, which is the situation of validation: the hypothesis was learned on the training set, and the validation set is a fresh sample. Setting the right-hand side to a tolerated failure probability $\delta$ and solving for $m$ turns a required accuracy $\epsilon$ and confidence $1 - \delta$ into a sample size:

\begin{equation}
\tag{3}
m \geq \frac{\ln(2/\delta)}{2 \epsilon^{2}} \text{,}
\end{equation}

and this many data points make the validation error lie within $\epsilon$ of the risk with probability at least $1 - \delta$ (Shalev-Shwartz and Ben-David, 2014, Thm. 11.1). For the frost warning, $\epsilon
= 0.05$ and $\delta = 0.05$ give $m \geq 738$ days; at that sample size all of $2000$ drawn samples (pythondemos/samplesize.py) lie within $0.05$ of the risk. The bound is loose, since it holds for every probability distribution and every loss function in $[0, 1]$; the sample size Eq. (3) is therefore enough, but not always needed. Its shape is what carries over: halving $\epsilon$ quadruples the sample size, while the sample size grows only logarithmically in $1/\delta$.

For training, the same question has a second answer. The hypothesis is then chosen by looking at the sample, so the bound Eq. (2) for a fixed hypothesis no longer applies to the learned one. The sample size then has to be compared with the size of the model. For a model of Vapnik–Chervonenkis dimension $d$, a sample size proportional to $d / \epsilon^{2}$ makes the training error of every hypothesis in the model lie within $\epsilon$ of its risk at once (Shalev-Shwartz and Ben-David, 2014, Thm. 6.8). A sample that is small against the effective dimension of the model is the typical cause of overfitting, and the learning curve shows the validation error of the learned hypothesis as a function of the sample size (Hastie et al., 2009, Sect. 7.10).

Sample size enters a third task, hypothesis testing: are two features of a data point correlated? For the days of 2024 at Krems, is the minimum temperature of a day correlated with its diurnal range, the maximum minus the minimum? Over the whole year, the Pearson correlation coefficient of the two features is $\rho = 0.16$. A test is given a sample of $m$ days and computes the sample correlation $r$, a function of sample means. Under the hypothesis that the two features are independent, $r$ scatters around $0$ with a standard deviation close to $1/\sqrt{m}$. For features that are Gaussian RVs, the variance of $r$ is exactly $1/(m-1)$ (Anderson, 2003, Thm. 4.2.1). For independent features with finite variances of any other probability distribution, $\sqrt{m}\, r$ converges to a standard Gaussian random variable by the central limit theorem applied to the products of the standardized features, whose variance is one under independence. For features that are Gaussian RVs, Fisher's transformation $\operatorname{artanh}(r)$ is approximately a Gaussian random variable with mean $\operatorname{artanh}(\rho)$ and standard deviation $1/\sqrt{m - 3}$ (Anderson, 2003, Thm. 4.2.5; Fisher, 1921). The test rejects ``uncorrelated'' when $|\operatorname{artanh}(r)| \sqrt{m - 3}$ exceeds the quantile $z_{1-\alpha/2}$ of the standard Gaussian random variable, which bounds the probability of a false rejection by the significance level $\alpha$. A correlation $\rho$ is then detected with probability at least $1 - \beta$, the power of the test, once

\begin{equation}
\tag{4}
m \geq 3 + \bigg( \frac{z_{1-\alpha/2} + z_{1-\beta}}
{\operatorname{artanh}(\rho)} \bigg)^{2} \text{.}
\end{equation}

For $\rho = 0.16$, $\alpha = 0.05$ ($z_{0.975} = 1.96$) and a power of $0.8$ ($z_{0.8} = 0.84$), the sample size Eq. (4) is $304$ days. Fig. 2 shows the power of the test against the sample size: $0.84$ at $m = 304$, and $0.10$ for a sample of $30$ days. The shape is that of Eq. (3) again: halving the correlation to be detected quadruples the sample size, so a weak correlation needs a large sample before ``uncorrelated'' can be rejected.

![Figure 2 of the entry samplesize](https://dictionaryofml.org/terms/fig/samplesize-fig2.svg)

*Figure 2: The power of the correlation test at significance level $\alpha = 0.05$ for the minimum temperature and the diurnal range of a day at Krems an der Donau, whose Pearson correlation coefficient over 2024 is $\rho = 0.16$, against the sample size (solid curve, the fraction of $1000$ samples of $m$ days on which the test rejects ``uncorrelated''). The dashed line marks the power $0.8$, the vertical line the sample size $304$ of Eq. (4). Data generated by pythondemos/samplesize.py*

The count $m$ is not by itself a measure of how much a sample carries. If it were, information could be created by copying: listing every day of a sample twice gives a sample of sample size $2m$ that says exactly what the original says. Its empirical distribution is unchanged, since each day now holds two entries of probability $1/(2m)$. So is the average loss Eq. (1) of every hypothesis, and so is the spread of that average over repeated samples. Only $2m$ days drawn afresh reduce that spread, by the factor $\sqrt{2}$. Read with $2m$ in place of $m$, the bound Eq. (2) would promise a deviation smaller by that same factor. That promise does not follow, since a copy of a day is not independent of the day it copies, while Hoeffding's inequality holds for realizations of independent and identically distributed RVs. The sample size counts the entries of a list, and how much they carry depends on how the list was generated.

Dependence is rarely as extreme as an outright copy. The sample sizes above assume independent and identically distributed RVs, and data points recorded in sequence, such as consecutive days, often show autocorrelation: a frost day is more likely after a frost day, and so is a wrong prediction. Dependent data points carry less information than independent ones. For a stationary sequence with autocorrelation $\rho_{\ell}$ at lag $\ell$, the variance of the average of $m$ consecutive values is that of $m$ independent values multiplied by

\begin{equation}
\tag{5}
1 + 2 \sum_{\ell=1}^{m-1}
\Big( 1 - \frac{\ell}{m} \Big) \rho_{\ell} \text{,}
\end{equation}

see Brockwell and Davis (1991, Thm. 7.1.1). The average is therefore as accurate as that of $m$ divided by Eq. (5) independent values, the effective sample size. For the 0/1 loss of the frost warning, the sample autocorrelation of consecutive days is $0.24$ at lag $1$ and stays positive for a week, and the factor Eq. (5) is $4.2$ for $m = 30$. The average over $30$ consecutive days therefore scatters as much as that over $7$ days drawn at random. A sample of consecutive data points must then be larger than Eq. (3) or Eq. (4) say, by the factor Eq. (5), or be drawn at random from the whole record, as the days of Fig. 1 are.

Dependence aside, the count says nothing about where the data points lie. Two samples of the same size can cover the feature space differently: one spread over the whole range of the features, the other crowded in a part of it. For a method whose prediction at a feature vector depends on the data points near it, the largest distance from a point of the feature space region of interest to the nearest data point governs the prediction error there, however large $m$ is (see covering). A weather record of summer days alone, however long, leaves the winter part of the feature space empty, and no sample size compensates for it.

**See also:** sample, [data point](https://dictionaryofml.org/terms/datapoint.html), [dataset](https://dictionaryofml.org/terms/dataset.html), [independent and identically distributed assumption](https://dictionaryofml.org/terms/iidasspt.html), risk, empirical risk, law of large numbers, concentration inequality, Hoeffding's inequality, validation, [validation error](https://dictionaryofml.org/terms/valerr.html), Vapnik–Chervonenkis dimension, learning curve, [overfitting](https://dictionaryofml.org/terms/overfitting.html), autocorrelation, empirical distribution, hypothesis testing, Pearson correlation coefficient.

## References

- Papoulis and Pillai (2002). Probability, Random Variables, and Stochastic Processes. *McGraw-Hill Higher Education*.
- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Anderson (2003). An Introduction to Multivariate Statistical Analysis. *Wiley*.
- Fisher (1921). On the ``probable error'' of a coefficient of correlation deduced from a small sample. *Metron*.
- Hoeffding (1963). Probability Inequalities for Sums of Bounded Random Variables. *J. Amer. Statistical Assoc.*. <https://doi.org/10.1080/01621459.1963.10500830>
- Brockwell and Davis (1991). Time Series: Theory and Methods. *Springer-Verlag*.

## Cite this entry

```bibtex
@misc{dictml_samplesize,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {sample size},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/samplesize.html}
}
```


---

# self-supervised learning

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/selfsupervisedlearning.html) · [typeset PDF](https://dictionaryofml.org/terms/selfsupervisedlearning.pdf) · [LaTeX](https://dictionaryofml.org/terms/selfsupervisedlearning.tex) · [Python demo](https://dictionaryofml.org/terms/selfsupervisedlearning.py) · [notebook](https://dictionaryofml.org/terms/selfsupervisedlearning.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Self-supervised learning constructs the labels of a training set from the data points themselves. Some features of a data point are withheld and used as its label while the rest remain features, so what results is an ordinary supervised learning problem, solved by empirical risk minimization on data that no annotator has touched. In natural language processing the withheld feature is the next token, which is how a large language model is trained. In computer vision it is a set of pixels, deleted from an image and predicted from the patches left visible.

## Definition

An open image database returns thousands of photographs for a keyword such as *cat*, and a web crawl yields billions of sentences. Neither comes with a label saying what should be predicted. A prediction task can be made from any one of them: hide the last word of a sentence, and the words before it become a question whose answer is already known. Self-supervised learning constructs labels this way, from the data point itself.

In general, some of the features of a data point are withheld and used as the label; the remaining ones stay as the features. The result is an ordinary supervised learning problem, solved by empirical risk minimization, and its training set needs nothing beyond the raw data. A single unlabeled collection yields as many training sets as there are ways of hiding part of a data point (see Fig. 1). The term goes back to de Sa (1993), who uses it for a classifier whose label for one sensory modality is derived from a co-occurring input in another.

The withheld feature can be a token. In natural language processing a data point is a stretch of text, its features are the $n$ tokens $v^{(1)}, \ldots, v^{(n)}$ of a context window of length $n$, and its label is the token $v^{(n+1)}$ that follows. Every position in every document is one labeled data point, so a corpus of $m$ tokens gives on the order of $m$ of them. This is the training task of a large language model such as GPT-3, an autoregressive language model, i.e., one trained to predict the next token (Brown et al., 2020, Sect. 1). Masking a token in the middle and predicting it from both sides withholds a different feature of the same data point (Devlin et al., 2019).

Similarly, in computer vision the withheld features are pixels: an image is split into patches, most of them are deleted, and the method predicts the missing pixel values from those that remain. He et al. (2022, Sect. 1 and Fig. 2) delete $75\%$ of the patches and still recover recognizable content.

A deep net solving either task computes a composition $\widehat{h} = s \circ \phi$. The feature transformation $\phi \colon \mathcal{X} \to \mathbb{R}^{d}$, with $\mathcal{X}$ the set of raw inputs (texts or images), sends a data point to a vector of activations, and the map $s$ predicts the withheld feature from that vector. That feature can be predicted only from what $\phi$ has kept: a next token cannot be recovered from a vector that has discarded the context, nor a deleted patch from one that has dropped its surroundings. Training the composition therefore constrains $\phi$ to keep that information, even though $\phi$ appears nowhere in the loss. The constraint does not determine $\phi$ uniquely: for any invertible map $g$ of $\mathbb{R}^{d}$ onto itself, the composition $(s \circ g^{-1}) \circ (g \circ \phi)$ equals $s \circ \phi$ and so has the same loss. He et al. (2022, Sect. 3) name the two halves: their encoder is $\phi$ and receives only the patches left visible, while their decoder is $s$ and rebuilds the pixels from its output. What restricts $\phi$ there is the deletion of most of the input, not a narrow layer as in an autoencoder.

In both domains it is $\phi$ that is carried over. The map $s$ is dropped, and a small replacement for it is fitted on the output of $\phi$ using the few labels the later task does have (see transfer learning, foundation model). Self-supervised learning is therefore one answer to the same shortage that motivates semi-supervised learning: there, unlabeled data points assist a label-scarce problem directly; here, they are turned into a labeled problem of their own first. Unsupervised learning is the third answer, and it gives up the label entirely: it finds structure among the feature vectors instead of predicting a withheld one.

![Figure 1 of the entry selfsupervisedlearning](https://dictionaryofml.org/terms/fig/selfsupervisedlearning-fig1.svg)

*Figure 1: Two ways of constructing labels from unlabeled data points, and in both a single source yields many. (a) Natural language processing: a sentence is read through a sliding window, and every position gives a labeled data point whose features are the tokens in the window and whose label is the token after it (shaded). (b) Computer vision: an image is split into patches and most are deleted (shaded); each missing patch, three of them outlined here, is one labeled data point whose label is its own pixel values. Neither task needs a human annotator*

**See also:** [feature](https://dictionaryofml.org/terms/feature.html), [label](https://dictionaryofml.org/terms/label.html), [supervised learning](https://dictionaryofml.org/terms/supervisedlearning.html), [unsupervised learning](https://dictionaryofml.org/terms/unsupervisedlearning.html), semi-supervised learning, [large language model](https://dictionaryofml.org/terms/llm.html), natural language processing, token, [transfer learning](https://dictionaryofml.org/terms/transferlearning.html), foundation model.

## References

- Brown et al. (2020). Language Models are Few-Shot Learners. *Adv. Neural Inf. Process. Syst.*. <https://proceedings.neurips.cc/paper_files/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html>
- Devlin et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. *Proc. 2019 Conf. North American Chapter Assoc. Computational Linguistics: Human Lang. Technologies, Volume 1 (Long and Short Papers)*. <https://doi.org/10.18653/v1/N19-1423>
- de Sa (1993). Learning Classification with Unlabeled Data. *Adv. Neural Inf. Process. Syst. (NIPS)*. <https://doi.org/10.5555/2987189.2987204>
- He et al. (2022). Masked Autoencoders Are Scalable Vision Learners. *2022 IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR)*. <https://doi.org/10.1109/CVPR52688.2022.01553>

## Cite this entry

```bibtex
@misc{dictml_selfsupervisedlearning,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {self-supervised learning},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/selfsupervisedlearning.html}
}
```


---

# sigmoid function

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/sigmoid.html) · [typeset PDF](https://dictionaryofml.org/terms/sigmoid.pdf) · [LaTeX](https://dictionaryofml.org/terms/sigmoid.tex) · [Python demo](https://dictionaryofml.org/terms/sigmoid.py) · [notebook](https://dictionaryofml.org/terms/sigmoid.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

The sigmoid function maps the real line into the interval between $0$ and $1$. It translates the real-valued score of a binary classifier into an estimate for the probability of a specific label value. It is smooth and strictly increasing, with a derivative that vanishes for an argument with large magnitude. Machine learning uses it in two roles: as an activation function inside a neuron, and as part of the log-likelihood function for learning the model parameters of a probabilistic model.

## Definition

A frost warning for Krems an der Donau has to say how likely frost is, not only whether it is expected. Fig. 1 shows minimum daytime temperatures measured at that station. A classifier that warns of frost reads the recent measurements of a day, its feature vector ${\mathbf x}$, and delivers a real-valued score $h({\mathbf x})$. The classification is a threshold: a frost warning for the next day is issued if and only if $h({\mathbf x}) > 0$, and the magnitude $|h({\mathbf x})|$ measures the confidence in that warning. In practice it is often more useful to state a probability. The sigmoid function translates the score $h({\mathbf x})$ into one, as Fig. 1 shows for its last day.

![Figure 1 of the entry sigmoid](https://dictionaryofml.org/terms/fig/sigmoid-fig1.svg)

*Figure 1: Six consecutive days at Krems an der Donau, each stem the lowest temperature measured that day. The measurements a classifier reads for the last of them are its feature vector ${\mathbf x}$, from which the classifier computes the score $h({\mathbf x}) = 0.26$. That score is positive, so a frost warning is issued, and the sigmoid function turns it into $\sigma(0.26) = 0.57$, a $57$ percent probability of frost on 17/01. Data generated by pythondemos/sigmoid.py*

The sigmoid function maps the real line into the interval between $0$ and $1$. It is the function $\sigma: \mathbb{R} \rightarrow (0,1)$ given by

$$
\sigma(z) := \frac{1}{1 + e^{-z}}
\text{.}
$$

It rises from $\sigma(z) \rightarrow 0$ as $z \rightarrow -\infty$ to $\sigma(z) \rightarrow 1$ as $z \rightarrow +\infty$, and passes through $\sigma(0) = 1/2$ (see Fig. 2). It is also called the logistic function, and it is the softmax function for two classes.

![Figure 2 of the entry sigmoid](https://dictionaryofml.org/terms/fig/sigmoid-fig2.svg)

*Figure 2: The sigmoid function $\sigma(z) = 1/(1 + e^{-z})$ maps the real line into the open interval $(0,1)$, with $\sigma(0) = 1/2$*

The sigmoid function is smooth and strictly increasing, and it satisfies the symmetry $\sigma(-z) = 1 - \sigma(z)$ (Bishop, 2006, Sect. 4.2). Its derivative is available in closed form, and in terms of the function itself (Bishop, 2006, Sect. 4.3.2, Eq. (4.88)),

$$
\sigma'(z) = \sigma(z)\big(1 - \sigma(z)\big)
\text{.}
$$

These properties make the sigmoid function a popular choice for the activation function of an artificial neural network. One drawback of that choice is that the derivative decreases quickly as the argument grows in magnitude, which slows the progress of gradient-based learning methods, the vanishing gradient problem. That is why the rectified linear unit replaced the sigmoid in the hidden layers of deep nets (Goodfellow et al., 2016, Sect. 6.3.2).

A probabilistic model for a binary label uses the sigmoid function to turn a score into a probability. Let the label $y$ be either $1$ or $-1$, and let the score be ${\mathbf w}^{\top} {\mathbf x}$ with model parameters ${\mathbf w}$. The model reads

$$
\mathbb{P}\left(y = 1 \mid {\mathbf x}\right)
= \sigma\big({\mathbf w}^{\top} {\mathbf x}\big)
\text{,} \qquad
\mathbb{P}\left(y = -1 \mid {\mathbf x}\right)
= \sigma\big(-{\mathbf w}^{\top} {\mathbf x}\big)
\text{,}
$$

where the second follows from the symmetry $\sigma(-z) = 1 - \sigma(z)$. Both cases are the single expression $\sigma(y \, {\mathbf w}^{\top} {\mathbf x})$, which is where the sigmoid function enters the log-likelihood. For a training set $\mathcal{D}^{(\rm train)} = \big\{ \left( {\mathbf x}^{(r)},y^{(r)} \right) \big\}_{r=1}^{m}$ of labeled data points, the log-likelihood of the model parameters is

$$
\sum_{r=1}^{m} \ln
\sigma\big( y^{(r)} {\mathbf w}^{\top}
{\mathbf x}^{(r)} \big)
\text{.}
$$

Maximizing it over ${\mathbf w}$ is maximum likelihood estimation, and it is logistic regression. Each summand is the negative of a logistic loss, so maximizing the log-likelihood and minimizing the average logistic loss are the same computation (Bishop, 2006, Sect. 4.3.2). For example, consider the classifier map $h$ learned by logistic regression from the days of $2024$ at Krems. Where its score is zero the sigmoid returns $1/2$, so the warning is issued exactly where the probability passes one half. A score of $2.14$ becomes a probability of $0.89$ for frost tomorrow, and a score of $-2.67$ becomes $0.07$.

**Synonyms:** logistic function.

**See also:** activation function, rectified linear unit, softmax function, [logistic regression](https://dictionaryofml.org/terms/logreg.html), linear model, classifier, probability, smooth.

## References

- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Goodfellow et al. (2016). Deep Learning. *MIT Press*.

## Cite this entry

```bibtex
@misc{dictml_sigmoid,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {sigmoid function},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/sigmoid.html}
}
```


---

# Sobolev space

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/sobolevspace.html) · [typeset PDF](https://dictionaryofml.org/terms/sobolevspace.pdf) · [LaTeX](https://dictionaryofml.org/terms/sobolevspace.tex) · [Python demo](https://dictionaryofml.org/terms/sobolevspace.py) · [notebook](https://dictionaryofml.org/terms/sobolevspace.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A Sobolev space is a set of functions whose derivatives, taken in a generalized sense, have bounded size. Machine learning methods whose hypothesis space is a ball in a Sobolev space with a bound on the gradient enforce a well-defined quantitative notion of robustness. Any hypothesis map in such a ball can be locally approximated by a linear map with bounded slope. A small change of the features of a data point therefore changes the prediction by at most a proportional amount. The generalized derivative is needed because hypothesis maps such as those of a deep net with rectified linear unit activation functions have kinks. A decision tree, whose hypothesis map jumps, belongs to no Sobolev space and admits no such guarantee. Linear regression whose hypothesis space is a ball in a Sobolev space, rather than the linear maps, is nonparametric regression. Sobolev norms also serve as regularizers, and they describe the reproducing kernel Hilbert space of a Matérn kernel.

## Definition

A weather service forecasts the maximum temperature of a day from its minimum temperature. Fig. 1(a) shows $20$ of the $366$ days of $2024$ at the station Krems an der Donau of GeoSphere Austria as the data points of a scatterplot, with the minimum temperature of a day as its feature and the maximum temperature as its label. The panel also shows three hypotheses learned from these data points: the non-linear map of a deep net with rectified linear unit activation functions, the step function of a depth-$2$ decision tree, and the curve of linear regression with a Sobolev penalty term. A trustworthy forecaster must not change its prediction, the forecast, abruptly: a thermometer error of half a degree in the minimum temperature must not move the forecast by several degrees. Sobolev spaces make this requirement precise. They consist of the functions whose derivatives have bounded size (Evans, 2010, Ch. 5). A bound on the derivative is a bound on how fast the forecast can change.

![Figure 1 of the entry sobolevspace](https://dictionaryofml.org/terms/fig/sobolevspace-fig1.svg)

*Figure 1: Forecasting the maximum temperature of a day at Krems an der Donau from its minimum temperature. (a) Scatterplot of every $19$th day of $2024$, $20$ of the $366$ days (circles), and three hypotheses learned from all $366$ days: the non-linear map of a deep net with rectified linear unit activation functions (solid curve), the step function of a depth-$2$ decision tree (dashed curve), and the curve of linear regression with a Sobolev penalty term (dotted curve). (b) The slope of the deep net hypothesis $\widehat{h}$ is a step function, its weak derivative, and stays below the bound $L = 2.44$ (dotted line), so a perturbation $\delta$ of the minimum temperature moves the forecast by at most $2.44\,|\delta|$. The tree has no such bound: at each of its thresholds (arrows) the forecast jumps. Data generated by pythondemos/sobolevspace.py*

The derivative is understood in a generalized sense, since the deep net of Fig. 1(a) has kinks and no classical derivative exists there. The generalized derivative starts from integration instead of differentiation. On the real line, a function with $f(x) = f(a) + \int_{a}^{x} g(t) \, \mathrm{d}t$ is the integral of its slope $g$, and $g$ is called a weak derivative of $f$. On an open set $\Omega \subseteq \mathbb{R}^{d}$, consider an infinitely differentiable test function $\varphi$ that vanishes outside a bounded subset of $\Omega$. A function $g_{j}$ is a weak partial derivative of $f$ if, for every such $\varphi$ (Evans, 2010, Sect. 5.2),

$$
\int_{\Omega} f({\mathbf x}) \frac{\partial \varphi({\mathbf x})}{\partial x_{j}} \, \mathrm{d}{\mathbf x}
= - \int_{\Omega} g_{j}({\mathbf x}) \varphi({\mathbf x}) \, \mathrm{d}{\mathbf x} \text{.}
$$

This is integration by parts with all derivatives moved onto $\varphi$. It holds when $f$ is the integral of $g_{j}$ along the $j$th coordinate, and it does not require a derivative of $f$ at any particular point. The slope of the deep net, the step function of Fig. 1(b), satisfies it for every test function tried (pythondemos/sobolevspace.py).

For $p \in [1,\infty]$, the Sobolev space $W^{1,p}(\Omega)$ consists of the functions whose weak derivatives exist and have finite $L^{p}$ norm $\left\lVert \cdot \right\rVert_{p}$. The case $p = 2$ gives the Hilbert space $H^{1}(\Omega)$, and the spaces $W^{k,p}(\Omega)$ require weak derivatives up to order $k$ (Adams and Fournier, 2003, Ch. 3, Thm. 3.6).

The case that matters most for machine learning is $W^{1,\infty}$, the functions with a bounded weak gradient. On a convex set $\Omega$, a function $f$ belongs to it with $\operatorname{ess\,sup}_{{\mathbf x} \in \Omega} \left\lVert \nabla f({\mathbf x}) \right\rVert_{2} \leq L$ precisely when it is Lipschitz continuous with constant $L$ (Evans, 2010, Sect. 5.8). Perturbing a feature vector by $\boldsymbol{\delta}$ then changes $f$ by at most $L \left\lVert \boldsymbol{\delta} \right\rVert_{2}$: convexity keeps the segment from ${\mathbf x}$ to ${\mathbf x} + \boldsymbol{\delta}$ inside $\Omega$, and integrating the gradient along it bounds the change by $L \left\lVert \boldsymbol{\delta} \right\rVert_{2}$. Conversely, the gradient of a Lipschitz continuous $f$ exists almost everywhere, and where it exists its length is the slope of $f$ in its own direction, at most $L$. The deep net of Fig. 1 has $L = 2.44$, so an error of $0.5\,^{\circ}$C in the minimum temperature moves its forecast by at most $1.22\,^{\circ}$C; the largest change among $20000$ random perturbations of that size was $1.21\,^{\circ}$C. The guarantee comes from the bound, not from the space: $W^{1,\infty}$ contains functions with every value of $L$. A machine learning method whose hypothesis space is the ball of functions with slope at most $L$ learns a robust hypothesis by construction, whatever the training set. The choice $p = \infty$ is what makes the bound hold at every feature vector; $H^{1}$ bounds the slope only on average. The bound concerns perturbations of the features only; robustness against perturbations of the training set is a property of the training map (see stability).

The bound also says that near almost every feature vector the hypothesis agrees with a linear map whose slope is at most $L$ (Rademacher's theorem, Evans, 2010, Sect. 5.8.3, Thm. 6). This linear map is the explanation that gradient-based explainable artificial intelligence delivers and that local interpretable model-agnostic explanations fits from perturbed data points. For the forecast, it reads: one additional degree in the minimum adds $s(x)$ degrees to the forecast, with the slope $s(x)$ of Fig. 1(b) between $0$ and $2.44$. Since the deep net is piecewise linear, the approximation is exact whenever no kink lies between $x$ and $x + \delta$ for a perturbation $\delta$ of the minimum temperature, which is the case for most pairs with $|\delta| \leq 0.1\,^{\circ}$C.

For the deep net, the weak gradient is bounded by the product of the spectral norms of its weight matrices, i.e., of their largest singular values (Tsuzuku et al., 2018, Sects. 5.1 and 5.2), here $46.0$ against the true $2.44$: valid but loose. For a classifier that takes the sign of such an $f$, the bound certifies that no perturbation of Euclidean norm below $|f({\mathbf x})|/L$ changes the prediction, since $|f({\mathbf x} + \boldsymbol{\delta}) - f({\mathbf x})| \leq L \left\lVert \boldsymbol{\delta} \right\rVert_{2} < |f({\mathbf x})|$ leaves the sign of $f$ unchanged (Tsuzuku et al., 2018, Prop. 2). The decision tree of Fig. 1(a) has no such bound: at its root threshold of $5.5\,^{\circ}$C, a change of $2 \cdot 10^{-9}\,^{\circ}$C in the minimum temperature moves the forecast by $7.81\,^{\circ}$C. A jump has no weak derivative that is a function (Evans, 2010, Sect. 5.2, Example 2), so no Sobolev space contains the tree.

Sobolev spaces also serve as hypothesis spaces in their own right. Plain linear regression minimizes the average squared error loss over the linear maps. Replacing the linear maps by a ball in $H^{1}$, or equivalently adding the squared $H^{1}$ seminorm $\int_{\Omega} \left\lVert \nabla f({\mathbf x}) \right\rVert_{2}^{2} \, \mathrm{d}{\mathbf x}$ as a penalty term, a norm that ignores constant shifts of $f$ (see regularization), gives nonparametric regression (Wainwright, 2019, Sects. 13.1.2 and 13.4). The dotted curve of Fig. 1(a) minimizes the training error plus ten times that seminorm; it reaches a training error of $18.03$, below the $18.66$ of the deep net, with slope at most $3.05$ (pythondemos/sobolevspace.py).

A computer cannot search all of $H^{1}$, and it need not. For a scalar feature, where $\Omega$ is an interval of the real line, a minimizer of the training error plus the seminorm is piecewise linear with kinks at the $m$ data points, since between two data points the straight line has the smallest seminorm for given end values (a consequence of the Cauchy–Schwarz inequality). The search therefore runs over $m$ numbers, the values at the kinks; this is kernel ridge regression with the reproducing kernel Hilbert space of $H^{1}$ (Wainwright, 2019, Sect. 12.5.2 and Example 12.16). For a piecewise linear function the seminorm is a sum of squared slopes, one per segment and weighted by its length, so the normal equations of the penalized problem form one linear system.

The seminorm expresses the smoothness assumption. Its counterpart for a function on the nodes of a graph replaces the gradient by differences across edges, the form used by generalized total variation and generalized total variation minimization (SarcheshmehPour et al., 2023); penalizing the gradient in $L^{1}$ instead gives total variation regularization, which allows jumps (Chambolle and Pock, 2016, Sect. 7.1; Rudin et al., 1992). The reproducing kernel Hilbert space of a Matérn kernel is a Sobolev space $W^{s,2}(\Omega)$ whose order $s$ is set by the smoothness parameter of the kernel (Kanagawa et al., 2018, Example 2.6). A kernel method with that kernel is therefore again nonparametric regression over a Sobolev ball, the setting of Wainwright (2019, Ch. 13) above.

Finally, a bound $L$ is also of regulatory interest. Article 15 of the EU AI Act requires a high-risk artificial intelligence system to be robust and to have its accuracy metrics declared in the instructions for use (European Parliament and Council of the European Union, 2024). A Lipschitz constant is such a metric: it states, as a single number, how much a perturbation of a feature vector can change the prediction.

**See also:** Lipschitz continuity, robustness, stability, deep net, [decision tree](https://dictionaryofml.org/terms/decisiontree.html), [linear regression](https://dictionaryofml.org/terms/linreg.html), [regularization](https://dictionaryofml.org/terms/regularization.html), [explainable artificial intelligence](https://dictionaryofml.org/terms/xaiterm.html), local interpretable model-agnostic explanations, smoothness assumption, generalized total variation, generalized total variation minimization, reproducing kernel Hilbert space, [Hilbert space](https://dictionaryofml.org/terms/hilbertspace.html), differentiable.

## References

- Evans (2010). Partial Differential Equations. *American Mathematical Society*.
- Adams and Fournier (2003). Sobolev Spaces. *Academic Press*.
- Tsuzuku et al. (2018). Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks. *Adv. Neural Inf. Process. Syst. (NeurIPS)*.
- Wainwright (2019). High-Dimensional Statistics: A Non-Asymptotic Viewpoint. *Cambridge University Press*.
- SarcheshmehPour et al. (2023). Clustered Federated Learning via Generalized Total Variation Minimization. *IEEE Trans. Signal Process.*. <https://doi.org/10.1109/TSP.2023.3322848>
- Chambolle and Pock (2016). An introduction to continuous optimization for imaging. *Acta Numer.*. <https://doi.org/10.1017/S096249291600009X>
- Rudin et al. (1992). Nonlinear total variation based noise removal algorithms. *Physica D: Nonlinear Phenomena*.
- Kanagawa et al. (2018). Gaussian Processes and Kernel Methods: A Review on Connections and Equivalences. <https://arxiv.org/abs/1807.02582>
- European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act) (Text with EEA relevance). <https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng>

## Cite this entry

```bibtex
@misc{dictml_sobolevspace,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {Sobolev space},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/sobolevspace.html}
}
```


---

# stochastic gradient descent (SGD)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-11. [HTML](https://dictionaryofml.org/terms/stochGD.html) · [typeset PDF](https://dictionaryofml.org/terms/stochGD.pdf) · [LaTeX](https://dictionaryofml.org/terms/stochGD.tex) · [Python demo](https://dictionaryofml.org/terms/stochGD.py) · [notebook](https://dictionaryofml.org/terms/stochGD.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Stochastic gradient descent (SGD) is a variant of gradient descent in which the gradient of the objective function is replaced by a computationally cheaper stochastic approximation. Its main application in machine learning is empirical risk minimization on a training set that is large or distributed: the gradient of the empirical risk averages over the entire training set. SGD approximates it by averaging over a randomly drawn batch of data points. The batch size trades the computational cost of a single update against the accuracy of the gradient approximation.

## Definition

Consider the weather stations of Fig. 1, each recording one data point: the minimum temperature of a day as feature and its maximum temperature as label. A weather forecasting application uses these data points to learn a hypothesis via empirical risk minimization. For a parametric model with model parameters ${\mathbf w}$ and a loss function $L\left({\mathbf z},{\mathbf w} \right)$, the objective function of empirical risk minimization can be written as

$$
f({\mathbf w}) := \frac{1}{m}
\sum_{r=1}^{m}
L\left({\mathbf z}^{(r)},{\mathbf w} \right) \text{.}
$$

This is the average loss over the training set $\mathcal{D}^{(\rm train)} = \big\{ {\mathbf z}^{(1)}, \ldots,
{\mathbf z}^{(m)} \big\}$ of $m$ data points. Note that minimizing $f$ on the training set is not the main goal in machine learning applications. The goal is to learn a hypothesis with a small loss on data points beyond the training set (see generalization).

For linear regression, which learns a linear hypothesis under the squared error loss, the above objective function $f$ is differentiable and convex. Gradient descent approaches a minimum of a differentiable function $f({\mathbf w})$ by repeating the gradient step

$$
{\mathbf w}^{(t+1)} = {\mathbf w}^{(t)}
- \eta \nabla f({\mathbf w}^{(t)})
\text{,}
$$

with a step size $\eta > 0$ that is small enough for each step to decrease $f$ (see gradient descent) (Nesterov, 2004, Thm. 2.1.14). The gradient of the above objective function is an average with one term per data point,

$$
\nabla f({\mathbf w}) = \frac{1}{m} \sum_{r=1}^{m} \nabla_{{\mathbf w}}
L\left({\mathbf z}^{(r)},{\mathbf w} \right).
$$

Its exact evaluation therefore requires collecting and processing the entire training set. This can be challenging or infeasible when the training set is large or stored at different locations. SGD approximates the gradient computation by replacing the sum over the entire training set with a sum over a subset of the training set. This subset is chosen randomly and referred to as a batch $\mathcal{B} \subseteq \{1, \ldots, m\}$ with batch size $B = |\mathcal{B}|$ (see Fig. 1). The resulting gradient approximation is

$$
{\mathbf g}^{(\mathcal{B})}\big( {\mathbf w} \big) := \frac{1}{B}
\sum_{r \in \mathcal{B}} \nabla_{{\mathbf w}}
L\left({\mathbf z}^{(r)},{\mathbf w} \right) \text{.}
$$

This gradient approximation can also be interpreted as the approximation of $f({\mathbf w})$ itself by the average loss over the batch (see Fig. 2).

![Figure 1 of the entry stochGD](https://dictionaryofml.org/terms/fig/stochGD-fig1.svg)

*Figure 1: A training set of $m = 12$ data points: each is the daily minimum temperature (feature $x$) and daily maximum temperature (label $y$) observed on 1 July 2026 at a different weather station (triangles), each its own device holding the single data point it recorded. The dotted vertical segments are the prediction errors of a linear hypothesis map $h$ (solid curve). A gradient descent update of the model parameters ${\mathbf w}$ uses the prediction errors of all $m$ data points; an SGD update uses only those of a randomly chosen batch $\mathcal{B}$ (inside red rings). Data generated by pythondemos/stochGD.py, which downloads the observations from the open data service of GeoSphere Austria*

![Figure 2 of the entry stochGD](https://dictionaryofml.org/terms/fig/stochGD-fig2.svg)

*Figure 2: SGD as gradient descent on an approximation of the objective function: the empirical risk $f({\mathbf w})$, the average loss over all $m$ data points of the training set (solid curve), and the average over one randomly drawn batch $\mathcal{B}$ (dashed curve). The gradient of the dashed curve is the stochastic estimate ${\mathbf g}({\mathbf w})$ that SGD uses in place of $\nabla f({\mathbf w})$*

Each SGD iteration generates a new batch $\mathcal{B}^{(t)}$ and performs the gradient descent update with ${\mathbf g}({\mathbf w})$ in place of the gradient $\nabla f({\mathbf w})$. The update reads

$$
{\mathbf w}^{(t+1)} = {\mathbf w}^{(t)}
- \eta^{(t)} \underbrace{ \frac{1}{B}
\sum_{r \in \mathcal{B}^{(t)}} \nabla_{{\mathbf w}}
L\left({\mathbf z}^{(r)},{\mathbf w}^{(t)} \right)}_{= {\mathbf g}^{(\mathcal{B}^{(t)})}\big( {\mathbf w}^{(t)} \big)}
\text{,}
$$

with a step size $\eta^{(t)} > 0$. Conditioned on ${\mathbf w}^{(t)}$, the batch $\mathcal{B}^{(t)}$ is drawn independently of it, so every data point enters with equal probability. The expectation of the gradient approximation ${\mathbf g}^{(\mathcal{B}^{(t)})}\big( {\mathbf w}^{(t)} \big)$ is therefore the exact gradient $\nabla f({\mathbf w}^{(t)})$: like any average over uniformly drawn data points, the approximation is an unbiased estimator (Bertsekas and Tsitsiklis, 2008, Sect. 9.1). The fixed points of the gradient step are the points satisfying the zero-gradient condition $\nabla f({\mathbf w}) = \mathbf{0}$. One classical set of sufficient conditions for the convergence of such random iterations to a minimum of $f$ is $\sum_{t=1}^{\infty} \eta^{(t)} = \infty$ and $\sum_{t=1}^{\infty} \big(\eta^{(t)}\big)^{2}
< \infty$ (Robbins and Monro, 1951, p. 404).

The batch size $B$ is the parameter of SGD that trades the computational cost of its iteration against the approximation error incurred in ${\mathbf g}^{(\mathcal{B})}\big( {\mathbf w} \big) \approx \nabla f({\mathbf w})$. One extreme case is $B = m$, i.e., the batch contains the entire training set and the gradient approximation becomes exact: ${\mathbf g}^{(\mathcal{B})}\big( {\mathbf w} \big) = \nabla f({\mathbf w})$. At $B = 1$, it approximates the gradient of $f$ using a single data point, resulting in a significant approximation error (see Fig. 3).

SGD with $1 < B < m$ is referred to as mini-batch SGD (Bottou, 1999). Each update costs $B$ gradient evaluations $\nabla L\left({\mathbf z}^{(r)},{\mathbf w}^{(t)} \right)$, for $r \in \mathcal{B}$, instead of $m$ evaluations required by computing the exact gradient. However, the update introduces an approximation error which decreases as the batch size grows (see Fig. 3).

![Figure 3 of the entry stochGD](https://dictionaryofml.org/terms/fig/stochGD-fig3.svg)

*Figure 3: The gradient approximation error $\Vert {{\mathbf g}^{(\mathcal{B})}\big( {\mathbf w} \big) - \nabla f({\mathbf w})} \Vert^{2}$ for the objective function $f$ of linear regression: the average squared error loss of a linear model over $m = 2000$ data points with $d = 5$ features. The approximation error, the filled circles, is plotted against the batch size $B$ and averaged over $300$ random choices of the batch. The gradient $\nabla f({\mathbf w})$ and its approximation ${\mathbf g}^{(\mathcal{B})}\big( {\mathbf w} \big)$ are taken at the same fixed model parameters ${\mathbf w} = \mathbf{0}$. The dashed reference is $v/B$, the squared error left by averaging $B$ independent terms of variance $v$, the variance of the gradient $\nabla_{{\mathbf w}} L\left({\mathbf z}^{(r)},{\mathbf w} \right)$ for a randomly chosen data point ${\mathbf z}^{(r)}$ (Bertsekas and Tsitsiklis, 2008, Sect. 9.1). The measurement follows that reference while $B$ is small against $m$, and falls below it as $B$ approaches $m$, because the batch is drawn without replacement: a batch covering much of the training set leaves few data points to differ between draws. The plot stops at $B = 1500$; at $B = m$ the approximation error would vanish, which the logarithmic axis cannot show. Data generated by pythondemos/stochGD.py*

Algorithm 1 states the method in full.

![Algorithm 1 of the entry stochGD](https://dictionaryofml.org/terms/fig/stochGD-alg1.svg)

*Algorithm 1: SGD*

Step 3 of Algorithm 1 evaluates $B$ gradients of the loss function, each of them $d$ numbers for a linear model, so one iteration costs on the order of $B d$ arithmetic operations against $m d$ for an exact gradient. That lower computational cost of SGD allows the training of high-dimensional models from large decentralized collections of data points (Bottou, 2010). A prime example of a machine learning application that involves large decentralized data is weather forecasting: the stations of Fig. 1, which open this entry, each hold a single data point, and a batch is a handful of stations polled for one gradient step.

**See also:** [gradient descent](https://dictionaryofml.org/terms/gd.html), gradient-based method, [gradient](https://dictionaryofml.org/terms/gradient.html), [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), batch, step size, objective function, zero-gradient condition, online gradient descent.

## References

- Bertsekas and Tsitsiklis (2008). Introduction to Probability. *Athena Scientific*.
- Nesterov (2004). Introductory Lectures on Convex Optimization: A Basic Course. *Kluwer Academic*. <https://doi.org/10.1007/978-1-4419-8853-9>
- Robbins and Monro (1951). A Stochastic Approximation Method. *The Annals of Mathematical Statistics*. <https://doi.org/10.1214/aoms/1177729586>
- Bottou (1999). On-line learning and stochastic approximations. *On-Line Learning in Neural Networks*.
- Bottou (2010). Large-scale machine learning with stochastic gradient descent. *Proceedings of COMPSTAT'2010*. <https://doi.org/10.1007/978-3-7908-2604-3_16>

## Cite this entry

```bibtex
@misc{dictml_stochGD,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {stochastic gradient descent (SGD)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/stochGD.html}
}
```


---

# supervised learning

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-10-28. [HTML](https://dictionaryofml.org/terms/supervisedlearning.html) · [typeset PDF](https://dictionaryofml.org/terms/supervisedlearning.pdf) · [LaTeX](https://dictionaryofml.org/terms/supervisedlearning.tex) · [Python demo](https://dictionaryofml.org/terms/supervisedlearning.py) · [notebook](https://dictionaryofml.org/terms/supervisedlearning.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Supervised learning is the machine learning setting that needs the label values of some data points. That requirement is what sets it apart from other forms of machine learning, such as unsupervised learning or reinforcement learning. Two main types of supervised learning are regression, for a numeric label, and classification, for a label with finitely many values. Empirical risk minimization learns a hypothesis of a hypothesis space that minimizes the average loss on a training set of labeled data points. A label can be obtained from a sensing device, computed, or annotated by a person. These labeling processes typically involve errors, so a recorded label value is a noisy estimate of the true one.

## Definition

A weather station of GeoSphere Austria at Krems an der Donau records the minimum and the maximum temperature of each day (GeoSphere Austria, 2026). From ten such days, two learning tasks can be built: predicting the maximum temperature of a day from its minimum, and predicting whether the next day is at least two degrees warmer. Both need, for each of the ten days, the value to be predicted, its label. Supervised learning refers to machine learning methods that need the label values of some data points, the ones forming a training set. That requirement sets it apart from the other settings: in unsupervised learning, no such value is available, and the loss is defined through model parameters that summarize the whole dataset; in reinforcement learning, a reward signal measures the usefulness of a prediction instead (Sutton and Barto, 2018, Sect. 1.1). Supervised learning methods evaluate the quality of a hypothesis by comparing its prediction against a known label value (Sutton and Barto, 2018, Sect. 1.1; Hastie et al., 2009, Ch. 2).

Empirical risk minimization learns a hypothesis by comparing its predictions for the data points in a training set with their actual labels. The training set $\mathcal{D}^{(\rm train)} = \big\{ \left( {\mathbf x}^{(r)},y^{(r)} \right) \big\}_{r=1}^{m}$ consists of $m$ labeled data points, each with a feature vector ${\mathbf x}^{(r)}$ and a label $y^{(r)}$. Among the hypotheses of a hypothesis space $\mathcal{H}$ it chooses one that minimizes the average loss on the training set,

$$
\widehat{h} \in \mathop{\mathrm{arg\,min}}_{h \in \mathcal{H}}
\frac{1}{m} \sum_{r=1}^{m}
L\left(\left( {\mathbf x}^{(r)},y^{(r)} \right),h \right)
\text{.}
$$

The loss of a data point can be evaluated on that data point alone, since its label supplies the value to compare with. The label space $\mathcal{Y}$, the set of values a label can take, decides which of two main types a learning task is: $\mathcal{Y} = \mathbb{R}$ makes it regression, a discrete label space makes it classification. Both learning tasks can be built from the same data points.

Fig. 1 shows two supervised learning tasks, one regression and one classification, obtained from the same data points, the ten days of Krems an der Donau. With the maximum temperature of the day as label, the learning task is regression, and empirical risk minimization with the squared error loss learns the curve of panel (a). The second learning task uses both temperatures of the day as features. Its label is $+1$ if the next day is at least two degrees warmer and $-1$ otherwise, so the learning task is classification. The two classes overlap, so no straight line separates them: the same morning can be followed by a warmer or by a cooler day.

![Figure 1 of the entry supervisedlearning](https://dictionaryofml.org/terms/fig/supervisedlearning-fig1.svg)

*Figure 1: Ten data points representing daily weather conditions at Krems an der Donau, the 25th of each month from January to October, are used in two different supervised learning tasks. In (a) the feature of a day is its minimum temperature and the label is its maximum temperature (open circles), so the learning task is regression; the curve is a hypothesis learned by empirical risk minimization with the squared error loss. In (b) both temperatures of a day are features and the label is $+1$ if the next day is at least two degrees warmer (filled circles) and $-1$ otherwise (open squares), so the learning task is classification. Data generated by pythondemos/supervisedlearning.py*

The label values of data points can be obtained in different ways, such as physical sensing devices, computations, or human annotation. For example, the maximum temperature of a day at Krems can be read from a calibrated thermometer of the weather station (GeoSphere Austria, 2026). The label value can also be obtained by numerical computations. GeoSphere Austria runs INCA, which computes the temperature across Austria on a grid of one kilometer (Haiden et al., 2010, Sect. 2; Haiden et al., 2011). INCA starts from the forecast of a numerical weather prediction model and corrects it with the differences between forecast and observation at the surrounding stations, so it returns a temperature value also for a location that has no station (Haiden et al., 2010, Sect. 5.1). The temperature recordings can be annotated by a person instead, a meteorologist or a panel of local residents.

These labeling processes differ in the amount of required resources and in the resulting label quality. Each carries an error of its own, so a recorded label value is a noisy estimate of the true one. The station thermometer has a calibration tolerance. A computed label inherits the error of the equations and of the readings they were given. A resident might misreport temperatures. Human annotation also limits how large the training set can be, since it requires the expert whose judgment is to be reproduced. Crowdsourcing lifts this limit: ImageNet, a dataset of $3.2$ million annotated images at the time of its first publication, was assembled by annotators working through a crowdsourcing platform (Deng et al., 2009, Sect. 3.2).

An annotated label records a judgment rather than a measurement, and two annotators need not return the same value. ImageNet was assembled to limit this disagreement. Several annotators vote on each image, which is kept only on a convincing majority. The number of votes required is fixed per category, from an initial subset of its images (Deng et al., 2009, Sect. 3.2).

**See also:** [label](https://dictionaryofml.org/terms/label.html), labeled data point, [training set](https://dictionaryofml.org/terms/trainset.html), [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), [hypothesis](https://dictionaryofml.org/terms/hypothesis.html), [loss](https://dictionaryofml.org/terms/loss.html), squared error loss, [classification](https://dictionaryofml.org/terms/classification.html), regression, classifier, [validation error](https://dictionaryofml.org/terms/valerr.html), [unsupervised learning](https://dictionaryofml.org/terms/unsupervisedlearning.html), [self-supervised learning](https://dictionaryofml.org/terms/selfsupervisedlearning.html), semi-supervised learning, [reinforcement learning](https://dictionaryofml.org/terms/reinforcementlearning.html).

## References

- GeoSphere Austria (2026). GeoSphere Austria climate station archive. <https://dataset.api.hub.geosphere.at/v1/station/historical/klima-v2-1d>
- Sutton and Barto (2018). Reinforcement Learning: An Introduction. *MIT Press*.
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Haiden et al. (2010). Integrated Nowcasting through Comprehensive Analysis (INCA) -- System description. <https://www.zamg.ac.at/fix/INCA_system.pdf>
- Haiden et al. (2011). The Integrated Nowcasting through Comprehensive Analysis (INCA) system and its validation over the Eastern Alpine region. *Weather and Forecasting*. <https://doi.org/10.1175/2010WAF2222451.1>
- Deng et al. (2009). ImageNet: A large-scale hierarchical image database. *2009 IEEE Conference on Computer Vision and Pattern Recognition*. <https://doi.org/10.1109/CVPR.2009.5206848>

## Cite this entry

```bibtex
@misc{dictml_supervisedlearning,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {supervised learning},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/supervisedlearning.html}
}
```


---

# singular value decomposition (SVD)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/svd.html) · [typeset PDF](https://dictionaryofml.org/terms/svd.pdf) · [LaTeX](https://dictionaryofml.org/terms/svd.tex) · [Python demo](https://dictionaryofml.org/terms/svd.py) · [notebook](https://dictionaryofml.org/terms/svd.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

The singular value decomposition (SVD) factors any matrix into an orthogonal matrix, a matrix that is zero off its main diagonal, and a second orthogonal matrix. For ${\mathbf A} \in \mathbb{R}^{m \times d}$ it reads ${\mathbf A} = \mathbf{V} \boldsymbol{\Lambda} \mathbf{U}^{\top}$, and the nonnegative diagonal entries of $\boldsymbol{\Lambda}$ are the singular values. In contrast to an eigenvalue decomposition, which only exists for a square diagonalizable matrix, an SVD exists for every matrix. Truncating the SVD after the $k$ largest singular values yields the best approximation by a matrix of rank $k$, which underlies dimensionality reduction. For linear regression, the SVD of the feature matrix splits the objective into one term per singular value plus an irreducible remainder. The solution with the smallest Euclidean norm divides each component of the label vector along a left singular vector by the matching singular value. The SVD delivers the pseudoinverse, the spectral norm, and the condition number of a matrix.

## Definition

A grayscale photograph with $m$ rows and $d$ columns of pixels is stored as a matrix of $m \times
d$ intensities, that is, $m d$ numbers. Most photographs can be stored with far fewer numbers: keeping only the $k$ largest of the singular values of that matrix, defined below, costs $k (m + d
+ 1)$ numbers. The SVD of a matrix ${\mathbf A} \in
\mathbb{R}^{m \times d}$ is a factorization of the form

$$
{\mathbf A} = \mathbf{V} {\boldsymbol \Lambda} \mathbf{U}^{\top} \text{,}
$$

with orthogonal matrices (Golub and Loan, 2013, Sect. 2.4), whose columns are orthonormal vectors,

$$
\mathbf{V} = \big({\mathbf v}^{(1)},\,\ldots,\,{\mathbf v}^{(m)}\big) \in \mathbb{R}^{m \times m},
\qquad
\mathbf{U} = \big( {\mathbf u}^{(1)},\,\ldots,\,{\mathbf u}^{(d)} \big) \in \mathbb{R}^{d \times d}
\text{.}
$$

The matrix ${\boldsymbol \Lambda} \in \mathbb{R}^{m \times d}$ is zero off the main diagonal, whose entries $\Lambda_{j,j} = \lambda_{j} \geq 0$ are referred to as singular values and ordered as $\lambda_{1} \geq \lambda_{2} \geq \ldots \geq 0$. The columns ${\mathbf u}^{(j)}$ of $\mathbf{U}$ are the right singular vectors and the columns ${\mathbf v}^{(j)}$ of $\mathbf{V}$ the left singular vectors; both families are orthonormal. Multiplying the factorization by ${\mathbf u}^{(j)}$ gives ${\mathbf A} {\mathbf u}^{(j)} = \lambda_{j} {\mathbf v}^{(j)}$. The matrix ${\mathbf A}$ therefore maps each right singular vector ${\mathbf u}^{(j)}$ to the left singular vector ${\mathbf v}^{(j)}$ scaled by the singular value $\lambda_{j}$ (see Fig. 1).

![Figure 1 of the entry svd](https://dictionaryofml.org/terms/fig/svd-fig1.svg)

*Figure 1: The matrix ${\mathbf A}$ maps the orthonormal vectors ${\mathbf u}^{(1)}, {\mathbf u}^{(2)}$ (left) to the orthogonal vectors $\lambda_{1} {\mathbf v}^{(1)}$ and $\lambda_{2} {\mathbf v}^{(2)}$ (right), scaled by the singular values*

In contrast to an eigenvalue decomposition, which only exists for square diagonalizable matrices, an SVD exists for every matrix (Golub and Loan, 2013, Sect. 2.4). The existence of the SVD and its computation follow from the spectral decomposition of the symmetric positive semi-definite matrix ${\mathbf A}^{\top}{\mathbf A}$. That matrix has an orthonormal basis of eigenvectors ${\mathbf u}^{(j)}$ of $\mathbb{R}^{d}$, with eigenvalues $\lambda_{j}^{2} \geq 0$, the squares of the singular values. The left singular vectors are then ${\mathbf v}^{(j)} := {\mathbf A} {\mathbf u}^{(j)} / \lambda_{j}$ for $\lambda_{j} > 0$, completed to an orthonormal basis of $\mathbb{R}^{m}$ for $\lambda_{j} = 0$. The matrices $\mathbf{U}$, ${\boldsymbol \Lambda}$, and $\mathbf{V}$ built this way satisfy ${\mathbf A} = \mathbf{V} {\boldsymbol \Lambda} \mathbf{U}^{\top}$. If ${\mathbf A}$ is itself symmetric and positive semi-definite, its SVD coincides with its eigenvalue decomposition. The largest singular value $\lambda_{1}$ equals the spectral norm $\left\lVert {\mathbf A} \right\rVert_{2}$ (Golub and Loan, 2013, Cor. 2.4.3), and, for a matrix of full rank, the ratio of the largest to the smallest singular value is the condition number $\kappa\!\left({\mathbf A}\right)$ (Golub and Loan, 2013, Sects. 2.6.2 and 5.3.1); a matrix of deficient rank has $\kappa\!\left({\mathbf A}\right) = \infty$ by convention.

Retaining only the $k$ largest singular values yields the truncated SVD

$$
\widehat{{\mathbf A}} := \sum_{j=1}^{k}
\lambda_{j} {\mathbf v}^{(j)}
\big({\mathbf u}^{(j)}\big)^{\top} \text{,}
$$

a matrix of rank at most $k$. Among all matrices ${\mathbf B}$ of rank at most $k$, the truncation $\widehat{{\mathbf A}}$ minimizes the approximation error $\left\lVert {\mathbf A} - {\mathbf B} \right\rVert_{\mathrm{F}}$ (Eckart and Young, 1936; Horn and Johnson, 2013, Sect. 7.4.9, Eq. (7.4.9.2)).

This property is what makes the truncation useful for storing the photograph of the opening. No matrix of rank at most $k$ is closer to ${\mathbf A}$, so the truncation loses as little as any storage by a rank-$k$ matrix can. It requires only the $k$ largest singular values with their left and right singular vectors, that is, $k (m + d + 1)$ numbers instead of $m d$ (pythondemos/svd.py).

Principal component analysis is the same truncation applied to a centered feature matrix ${\mathbf X} \in \mathbb{R}^{m \times d}$: a linear dimensionality reduction that replaces each feature vector by its coordinates along the $k$ directions of largest variance. The right singular vectors ${\mathbf u}^{(j)}$ of ${\mathbf X}$ are the eigenvectors of the sample covariance matrix $\frac{1}{m} {\mathbf X}^{\top}
{\mathbf X}$, whose eigenvalues are $\lambda_{j}^{2}/m$. Principal component analysis therefore projects each data point onto the leading ${\mathbf u}^{(1)},\,\ldots,\,{\mathbf u}^{(k)}$.

The SVD can also be used to analyze and to implement linear regression methods. Consider a feature matrix ${\mathbf X} \in \mathbb{R}^{m \times d}$, whose $r$-th row is the feature vector of the $r$-th data point, and a label vector ${\mathbf y} \in \mathbb{R}^{m}$. Linear regression minimizes the squared Euclidean norm $\left\lVert {\mathbf X} {\mathbf w} - {\mathbf y} \right\rVert_{2}^{2}$ of the residual over the model parameters ${\mathbf w}$. That is the empirical risk minimization objective for the squared error loss, the least squares problem.

Let $p$ be the number of nonzero singular values of ${\mathbf X}$, its rank. A component of ${\mathbf w}$ along a right singular vector ${\mathbf u}^{(j)}$ with $j > p$ leaves ${\mathbf X} {\mathbf w}$ unchanged, so expanding ${\mathbf w} = \sum_{j=1}^{p}
\widetilde{w}^{(j)} {\mathbf u}^{(j)}$ in the leading right singular vectors loses nothing. With ${\mathbf X} {\mathbf u}^{(j)} = \lambda_{j}{\mathbf v}^{(j)}$ and the orthonormality of the left singular vectors ${\mathbf v}^{(j)}$, the objective of linear regression reads

$$
\begin{aligned}
\left\lVert {\mathbf X} {\mathbf w} - {\mathbf y} \right\rVert_{2}^{2}
&= \left\lVert \textstyle\sum_{j=1}^{p}
\lambda_{j}\widetilde{w}^{(j)} {\mathbf v}^{(j)}
- {\mathbf y} \right\rVert_{2}^{2} \\
&= \sum_{j=1}^{p}
\big(\lambda_{j}\widetilde{w}^{(j)}
- ({\mathbf v}^{(j)})^{\top}{\mathbf y}\big)^{2}
+ \sum_{j=p+1}^{m}
\big(({\mathbf v}^{(j)})^{\top}{\mathbf y}\big)^{2} \text{.}
\end{aligned}
$$

The objective of linear regression depends on ${\mathbf y}$ only through its components $({\mathbf v}^{(j)})^{\top}{\mathbf y}$ along the left singular vectors. The second sum is the constant, irreducible error from the part of ${\mathbf y}$ outside the column space of ${\mathbf X}$. The first sum is minimized termwise at $\widehat{w}^{(j)} =
({\mathbf v}^{(j)})^{\top}{\mathbf y} / \lambda_{j}$ for $j \leq p$. In matrix form, this is $\widehat{{\mathbf w}} = {{\mathbf X}}^{+} {\mathbf y}$ with the pseudoinverse ${{\mathbf X}}^{+} = \mathbf{U} {{\boldsymbol
\Lambda}}^{+} \mathbf{V}^{\top}$, where ${{\boldsymbol \Lambda}}^{+}$ inverts the nonzero singular values. The vector $\widehat{{\mathbf w}}$ is the solution of linear regression with the smallest Euclidean norm (Golub and Loan, 2013, Thm. 5.5.1). A small singular value divides the corresponding component of ${\mathbf y}$ by a small number, which is why the condition number of ${\mathbf X}$ determines how noise in ${\mathbf y}$ enters $\widehat{{\mathbf w}}$ (Golub and Loan, 2013, Thm. 5.3.1). That matters because the least squares objective is a proxy: the goal is a small squared error on data points outside the training set (see generalization), and noise amplified into $\widehat{{\mathbf w}}$ is carried to every such data point.

**See also:** [matrix](https://dictionaryofml.org/terms/matrix.html), orthogonal, orthonormal, singular value, [eigenvalue decomposition](https://dictionaryofml.org/terms/evd.html), spectral norm, condition number, rank, pseudoinverse, [principal component analysis](https://dictionaryofml.org/terms/pca.html), [dimensionality reduction](https://dictionaryofml.org/terms/dimred.html), [linear regression](https://dictionaryofml.org/terms/linreg.html).

## References

- Golub and Loan (2013). Matrix Computations. *The Johns Hopkins Univ. Press*.
- Eckart and Young (1936). The approximation of one matrix by another of lower rank. *Psychometrika*. <https://doi.org/10.1007/BF02288367>
- Horn and Johnson (2013). Matrix Analysis. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781139020411>

## Cite this entry

```bibtex
@misc{dictml_svd,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {singular value decomposition (SVD)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/svd.html}
}
```


---

# support vector machine (SVM)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/svm.html) · [typeset PDF](https://dictionaryofml.org/terms/svm.pdf) · [LaTeX](https://dictionaryofml.org/terms/svm.tex) · [Python demo](https://dictionaryofml.org/terms/svm.py) · [notebook](https://dictionaryofml.org/terms/svm.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

The support vector machine (SVM) is a binary classification method that learns a linear classifier by regularized empirical risk minimization, combining the hinge loss with a squared-norm penalty term. For a linearly separable training set and sufficiently weak regularization, the solution is the maximum-margin separating hyperplane. This hyperplane is fully determined by the feature vectors closest to it, which are called support vectors. The kernel SVM is obtained by combining the basic SVM with a feature transformation derived from a kernel function.

## Definition

A handwritten digit is recognized from the pixel values of its image, and a text document is sorted by topic from the frequencies of its words (Cristianini and Shawe-Taylor, 2000). In both tasks a numeric feature vector has to be mapped to a label, and the SVM has been applied to both. The SVM is a binary classification method for data points with feature space $\mathcal{X}=\mathbb{R}^{d}$. In its simplest form, the SVM learns a linear classifier with decision boundary

$$
\{ {\mathbf x} \in \mathbb{R}^{d} : {\mathbf w}^{\top} {\mathbf x} + b = 0 \}.
$$

The linear classifier is parameterized by a nonzero vector ${\mathbf w} \in \mathbb{R}^{d} \setminus \{\mathbf{0}\}$ and an offset $b \in \mathbb{R}$. The vector ${\mathbf w}$ is the normal vector to the decision boundary and the offset $b$ shifts the decision boundary away from the origin. For a data point with feature vector ${\mathbf x}$ and label $y \in \{-1, +1\}$, the SVM delivers the prediction $\widehat{y} = \operatorname{sign}({\mathbf w}^{\top}
{\mathbf x} + b)$.

The SVM can be formulated as an instance of regularized empirical risk minimization for learning a linear classifier from a training set that consists of $m$ data points. The model parameters $(\widehat{{\mathbf w}},
\widehat{b})$ of the linear classifier are obtained as

\begin{equation}
\tag{1}
(\widehat{{\mathbf w}}, \widehat{b}) = \mathop{\mathrm{arg\,min}}_{{\mathbf w} \in \mathbb{R}^{d},\, b \in \mathbb{R}}
\underbrace{\frac{1}{m} \sum_{r=1}^{m}
\max\{0,\, 1 - y^{(r)} ({\mathbf w}^{\top} {\mathbf x}^{(r)} + b)\}
+ \alpha \left\lVert {\mathbf w} \right\rVert_{2}^{2}}_{:= f({\mathbf w}, b)}\text{.}
\end{equation}

The SVM objective function consists of the average hinge loss on the training set plus a penalty term $\alpha \left\lVert {\mathbf w} \right\rVert_{2}^{2}$ with regularization parameter $\alpha > 0$ (Hastie et al., 2009, Sect. 12.3.2, p. 426; Shalev-Shwartz and Ben-David, 2014, Sect. 15.2, Eq. (15.5)). The penalty term in the SVM problem Eq. (1) is the scaled squared Euclidean norm $\alpha \left\lVert {\mathbf w} \right\rVert_{2}^{2}$, the same penalty term as in ridge regression. It does not involve the model parameter $b$, which is therefore left unpenalized.

In linear regression, the penalty term estimates how much higher the loss is on data points outside the training set than on it (Hastie et al., 2009, Sect. 7.4, Eq. (7.24); see linear regression), an interpretation that requires the quadratic structure of the squared error loss. For the SVM, the penalty term instead controls the generalization gap: the hinge loss is Lipschitz continuous in ${\mathbf w}$, with a constant given by the Euclidean norm of the feature vector. Adding the penalty term $\alpha \left\lVert {\mathbf w} \right\rVert_{2}^{2}$ to the average hinge loss has three effects: First, it makes the overall objective function in the SVM problem Eq. (1) strongly convex in ${\mathbf w}$, so that its solution $\widehat{{\mathbf w}}$ is unique (Shalev-Shwartz and Ben-David, 2014, Lemma 13.5). Second, it controls how much that solution varies if the data points in the training set change (Shalev-Shwartz and Ben-David, 2014, Cor. 13.6). Third, it keeps $\left\lVert {\mathbf w} \right\rVert_{2}$ small and therefore the margins wide (Shalev-Shwartz and Ben-David, 2014, Sect. 15.2).

If the feature vectors in the training set have bounded Euclidean norm, the expectation of the difference between the risk and the empirical hinge loss of the learned linear classifier decreases in proportion to $1/(\alpha m)$ (Shalev-Shwartz and Ben-David, 2014, Cor. 13.6, Cor. 15.7).

The solution of the SVM problem Eq. (1) has a geometric meaning when the training set is linearly separable, i.e., there is some choice for $({\mathbf w}, b)$ such that $\operatorname{sign}({\mathbf w}^{\top} {\mathbf x}^{(r)}
+ b) y^{(r)}=1$ for $r=1, \ldots, m$. For a sufficiently small $\alpha$, the solution of the SVM problem Eq. (1) determines the separating hyperplane that is farthest from the feature vectors in the training set (see Fig. 1). The distance between this hyperplane and the nearest feature vectors, which are precisely the support vectors discussed below, is referred to as the margin and is given by $1/\left\lVert \widehat{{\mathbf w}} \right\rVert_{2}$ (Boser et al., 1992; Cortes and Vapnik, 1995; Cristianini and Shawe-Taylor, 2000; Hastie et al., 2009, Sect. 12.2, Eq. (12.4)).

![Figure 1 of the entry svm](https://dictionaryofml.org/terms/fig/svm-fig1.svg)

*Figure 1: Maximum-margin separation for a linearly separable training set. The decision boundary $\widehat{{\mathbf w}}^{\top}{\mathbf x} + \widehat{b} = 0$ (solid line) lies halfway between two parallel hyperplanes $\widehat{{\mathbf w}}^{\top}{\mathbf x} + \widehat{b} = \pm 1$ (dashed lines). The support vectors (open square and filled circle inside red rings) are the feature vectors that lie on these hyperplanes. The margin $\gamma = 1/\left\lVert \widehat{{\mathbf w}} \right\rVert_{2}$ is the distance from the decision boundary to the support vectors. Data points with label $y=+1$ are drawn as filled circles, those with $y=-1$ as open squares. The dashed circle of radius $\gamma$ shows that any support vector can be displaced by a perturbation with Euclidean norm less than $\gamma$ without crossing the decision boundary*

The margin of the learned hyperplane measures robustness against perturbations of the features of a data point. Perturbing a feature vector ${\mathbf x}$ by a vector $\boldsymbol{\delta}$ changes the score $\widehat{{\mathbf w}}^{\top}{\mathbf x} + \widehat{b}$ by $\widehat{{\mathbf w}}^{\top}\boldsymbol{\delta}$, whose magnitude is at most $\left\lVert \widehat{{\mathbf w}} \right\rVert_{2} \left\lVert \boldsymbol{\delta} \right\rVert_{2}$ by the Cauchy-Schwarz inequality. A data point therefore stays correctly classified under every feature perturbation with $\left\lVert \boldsymbol{\delta} \right\rVert_{2} < \gamma$, as illustrated by the dashed circle around a support vector in Fig. 1. Data points whose feature vectors lie farther from the decision boundary tolerate even larger perturbations: the tolerated radius equals the distance of the feature vector from the decision boundary, which is at least $\gamma$ and equals $\gamma$ precisely for the support vectors. Maximizing the margin maximizes this worst-case tolerated radius, so the maximum-margin classifier is the separating classifier that is robust against the largest feature perturbations. The same holds beyond the separable case: the SVM problem Eq. (1) is equivalent to minimizing the hinge loss under the worst case over bounded perturbations of the feature vectors (Xu et al., 2009, Thm. 3).

Every solution of the SVM problem Eq. (1) has a structure that holds whether or not the training set is linearly separable. In particular,

$$
\widehat{{\mathbf w}} = \sum_{r=1}^{m} \beta_{r} y^{(r)} {\mathbf x}^{(r)}
$$

with expansion coefficients $0 \leq \beta_{r} \leq 1/(2 \alpha m)$ that are nonzero only for the feature vectors with

$$
y^{(r)} (\widehat{{\mathbf w}}^{\top} {\mathbf x}^{(r)} + \widehat{b}) \leq 1.
$$

These feature vectors are referred to as support vectors since they entirely determine the SVM solution $(\widehat{{\mathbf w}}, \widehat{b})$ (Boser et al., 1992; Cortes and Vapnik, 1995, Sect. 2). Applying a perturbation to any feature vector that is not a support vector leaves $\widehat{{\mathbf w}}$, $\widehat{b}$ unchanged, provided the perturbation is small enough to preserve the strict inequality $y (\widehat{{\mathbf w}}^{\top} {\mathbf x} + \widehat{b}) > 1$ (see Fig. 2, whose training set contains an outlier).

![Figure 2 of the entry svm](https://dictionaryofml.org/terms/fig/svm-fig2.svg)

*Figure 2: Non-separable training set of $m = 7$ data points in feature space $\mathbb{R}^{2}$. The solution of the SVM problem Eq. (1) places the decision boundary $\widehat{{\mathbf w}}^{\top}{\mathbf x} + \widehat{b} = 0$ (solid line) between the two hyperplanes $\widehat{{\mathbf w}}^{\top}{\mathbf x} + \widehat{b} = \pm 1$ (dashed lines) at distance $\gamma = 1/\left\lVert \widehat{{\mathbf w}} \right\rVert_{2}$. These two hyperplanes contain four of the support vectors (red rings), given by the data points with margin equal to $\gamma$. The outlier, labeled $+1$, is also a support vector; it is misclassified beyond the opposite margin: $y(\widehat{{\mathbf w}}^{\top}{\mathbf x} + \widehat{b}) = -2 < -1$, with hinge loss value $\xi$. The two data points (open square and filled circle not inside red rings) farther from the decision boundary than the margin $\gamma$ are not support vectors. Data generated by pythondemos/svm.py*

Like logistic regression and linear regression, the SVM learns the model parameters $({\mathbf w}, b)$ of a linear model. In contrast to logistic regression and linear regression, the SVM objective function $f({\mathbf w}, b)$ is non-smooth because of the hinge loss. However, the hinge loss is the maximum of two affine functions of $({\mathbf w}, b)$ and therefore convex (Boyd and Vandenberghe, 2004, Sect. 3.2.3), so the SVM objective function, the sum of its average and the convex penalty term, is convex and the optimization problem Eq. (1) can be solved by convex optimization methods. One such method is subgradient descent, which is obtained from gradient descent by replacing the gradient of the objective function with a subgradient (Boyd and Vandenberghe, 2004; Polyak, 1987, Sect. 5.3).

Starting from an initial choice ${\mathbf w}^{(0)}, b^{(0)}$ of the model parameters, subgradient descent repeatedly applies the update

$$
\big({\mathbf w}^{(t+1)}, b^{(t+1)}\big)
= \big({\mathbf w}^{(t)}, b^{(t)}\big)
- \eta^{(t)} {\mathbf g}^{(t)}
\text{, for } t = 0,1,\ldots \text{,}
$$

with a step size $\eta^{(t)} > 0$ and a subgradient ${\mathbf g}^{(t)} \in \partial f\big({\mathbf w}^{(t)}, b^{(t)}\big)$ of the SVM objective function Eq. (1). Inserting the SVM objective function Eq. (1) into the generic update yields the explicit update

$$
\begin{aligned}
{\mathbf w}^{(t+1)} &= \big(1 - 2 \eta^{(t)} \alpha\big) {\mathbf w}^{(t)}
+ \frac{\eta^{(t)}}{m}
\sum_{r=1}^{m} \beta^{(t)}_{r}
y^{(r)} {\mathbf x}^{(r)} \text{,} \\
b^{(t+1)} &= b^{(t)}
+ \frac{\eta^{(t)}}{m}
\sum_{r=1}^{m} \beta^{(t)}_{r}
y^{(r)} \text{,}
\end{aligned}
$$

with expansion coefficients $\beta^{(t)}_{r} \in [0, 1]$, for $r = 1, \ldots, m$. The coefficient $\beta^{(t)}_{r}$ is nonzero only for data points that satisfy $y^{(r)} \big( \big({\mathbf w}^{(t)}\big)^{\top} {\mathbf x}^{(r)} + b^{(t)} \big) \leq 1$, i.e., data points on which the hinge loss is nonzero or non-differentiable at the current model parameters. Each iteration scales ${\mathbf w}^{(t)}$ by the factor $1 - 2 \eta^{(t)} \alpha$ (the effect of the penalty term) and adds corrections $y^{(r)} {\mathbf x}^{(r)}$ only from these data points.

In particular, for the initialization ${\mathbf w}^{(0)} = \mathbf{0}$, every iterate ${\mathbf w}^{(t)}$ is a weighted sum $\sum_{r=1}^{m} \beta_{r} y^{(r)} {\mathbf x}^{(r)}$ of the feature vectors in the training set. This has the form of the expansion of the solution $\widehat{{\mathbf w}}$ in the support vectors stated above, built up one iteration at a time. A coefficient of the iterate ${\mathbf w}^{(t)}$ is nonzero only for a data point that satisfied $y^{(r)} \big( \big({\mathbf w}^{(t')}\big)^{\top} {\mathbf x}^{(r)} + b^{(t')} \big) \leq 1$ at some earlier iteration $t' < t$. For step sizes with $2 \eta^{(t')} \alpha < 1$, such a coefficient stays positive afterwards, since every later iteration rescales it by a positive factor and adds a nonnegative amount. The iterate can therefore also carry feature vectors that are not support vectors of the solution $\widehat{{\mathbf w}}$. For a constant step size $\eta = B/(\rho\sqrt{T})$ over $T$ iterations, where $B$ bounds the norm of a minimizer of Eq. (1) and $\rho$ the norm of the subgradients used, the SVM objective function at the average of the iterates exceeds its minimum by at most $B\rho/\sqrt{T}$ (Shalev-Shwartz and Ben-David, 2014, Thm. 14.8).

An overly large regularization parameter $\alpha$ makes the SVM underfit: the above expansion of $\widehat{{\mathbf w}}$ implies $\left\lVert \widehat{{\mathbf w}} \right\rVert_{2} \leq \rho/(2\alpha)$, with $\rho$ the largest Euclidean norm of a feature vector in the training set, so $\widehat{{\mathbf w}}$ shrinks toward $\mathbf{0}$. Consider a training set with unequal class sizes. For sufficiently large $\alpha$, the predictions approach the constant majority-class prediction $\operatorname{sign}(\widehat{b})$ and every data point of the minority class is a misclassified support vector. The training set of Fig. 2 also has unequal class sizes, four data points labeled $y = +1$ and three labeled $y = -1$, but with the value $\alpha = 1/14$ used there, the SVM misclassifies one data point that can be considered an outlier. Increasing the value of $\alpha$ to $50$ for that training set shrinks $\left\lVert \widehat{{\mathbf w}} \right\rVert_{2}$, makes every prediction on the training set equal to $+1$, and misclassifies all three data points of the minority class.

The basic SVM Eq. (1) learns a linear classifier on the feature space $\mathcal{X} = \mathbb{R}^{d}$. It can be generalized to a classification method for data points whose feature vectors lie in an arbitrary feature space $\mathcal{X}$ by first applying a feature transformation $\phi: \mathcal{X} \to \mathcal{X}'$ with $\mathcal{X}' = \mathbb{R}^{d}$ and then learning a linear classifier on the transformed feature vectors. By choosing $\phi$ (and $d$) appropriately, any given training set with distinct feature vectors can be made linearly separable in $\mathcal{X}'$, e.g., by mapping each distinct feature vector to its own coordinate axis (Cristianini and Shawe-Taylor, 2000). A general construction of a feature transformation $\phi$ is via a kernel $k: \mathcal{X} \times \mathcal{X} \to \mathbb{R}$. In this construction, the transformed feature space is, in general, not $\mathbb{R}^{d}$ but a Hilbert space $\mathcal{H}$, which can be infinite-dimensional (see kernel method). The resulting method is then referred to as a kernel SVM (Schölkopf and Smola, 2002).

**Synonyms:** maximum-margin classifier.

**See also:** [binary classification](https://dictionaryofml.org/terms/binclass.html), hinge loss, margin, hyperplane, [decision boundary](https://dictionaryofml.org/terms/decisionboundary.html), robustness, [kernel method](https://dictionaryofml.org/terms/kernelmethod.html), subgradient descent.

## References

- Cristianini and Shawe-Taylor (2000). An Introduction to Support Vector Machines and Other Kernel-based Learning Methods. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9780511801389>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Boser et al. (1992). A Training Algorithm for Optimal Margin Classifiers. *Proc. 5th Annual Workshop on Computational Learning Theory (COLT)*. <https://doi.org/10.1145/130385.130401>
- Cortes and Vapnik (1995). Support-vector networks. *Machine learning*. <https://doi.org/10.1007/BF00994018>
- Xu et al. (2009). Robustness and Regularization of Support Vector Machines. *J. Mach. Learn. Res.*. <https://www.jmlr.org/papers/v10/xu09b.html>
- Boyd and Vandenberghe (2004). Convex Optimization. *Cambridge Univ. Press*. <https://doi.org/10.1017/CBO9780511804441>
- Polyak (1987). Introduction to optimization. *Optimization Software, Inc., Publications Division, New York*.
- Schölkopf and Smola (2002). Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. *MIT Press*. <https://doi.org/10.7551/mitpress/4175.001.0001>

## Cite this entry

```bibtex
@misc{dictml_svm,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {support vector machine (SVM)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/svm.html}
}
```


---

# tabular data

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-06. [HTML](https://dictionaryofml.org/terms/tabulardata.html) · [typeset PDF](https://dictionaryofml.org/terms/tabulardata.pdf) · [LaTeX](https://dictionaryofml.org/terms/tabulardata.tex) · [Python demo](https://dictionaryofml.org/terms/tabulardata.py) · [notebook](https://dictionaryofml.org/terms/tabulardata.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Tabular data consist of data points that share a common, fixed set of attributes. The measured attributes serve as the features or as the label of a data point. The data points constitute the rows, or records, of a table, and the attributes are its columns. Every row has one cell per column, and the cells within one column hold values of the same attribute. The fixed attribute set separates tabular data from other types of data, such as text or graphs, whose data points carry no common set of attributes.

## Definition

A Finnish Meteorological Institute weather station measures the weather of each day, and the measurements are represented as a table with one row for each day (Fig. 1(a)). The table has four columns, one per attribute of a day: the date, the morning minimum temperature, the precipitation, and the maximum daytime temperature (Finnish Meteorological Institute, 2024). The date identifies the data point, and the three measured attributes are the candidates for its features and its label. Tabular data consist of data points that share such a common, fixed set of attributes (Everitt, 2002). As the name suggests, tabular data can be stored as a table: the rows of the table are the data points, and the columns of the table are the attributes.

In Fig. 1(b), the learning task is to forecast the maximum daytime temperature of a day from its morning minimum temperature. A machine learning method reads the minimum-temperature column as the feature and the maximum-temperature column as the label, while the precipitation column is left unused. It learns a hypothesis from the $366$ data points, shown as a scatterplot with the learned curve.

![Figure 1 of the entry tabulardata](https://dictionaryofml.org/terms/fig/tabulardata-fig1.svg)

*Figure 1: (a) Four of the $366$ rows of the table of daily weather measurements recorded in $2024$ at the Finnish Meteorological Institute station Helsinki Kaisaniemi (January 8–11). Each row is a data point, namely one day. The headers abbreviate the morning minimum temperature (min. temp.), the precipitation (precip.), and the maximum daytime temperature (max. temp.). Every row has one cell per column. The two shaded columns are the ones used in panel (b), and the braces above them name the roles: the light gray column holds the feature and the dark gray column the label of a day. (b) Scatterplot of $20$ of the $366$ days (every $19$th), with the minimum temperature of a day as its feature and the maximum temperature as its label; the curve depicts a hypothesis learned from all $366$ data points. Which columns serve as feature and label of a learning task is a design choice, and the panel shows one such choice. Data generated by pythondemos/tabulardata.py*

The distinct characteristic of tabular data is the rigid shape of a table. Every row contains the same number of cells, one per column, and the cells within one column hold values of the same attribute of the data points. The value range must be defined for each column, and every cell of a column holds a value from that range. The precipitation column of Fig. 1(a), for example, holds a daily amount in millimeters or the special value $-1.0$, which records that no precipitation was observed. A second, different use of a special value is a marker such as ``N/A'' reserved for an empty cell (Tennison et al., 2015). Missing data are represented by an empty cell: an attribute that was not recorded for a data point leaves its cell empty, while the table keeps its shape. Data imputation fills such a cell with an estimate computed from the cells that were recorded.

The fixed attribute set is what separates tabular data from other data types. For the weather measurements, the station records the same quantities every day, so the attribute set is fixed by the measurement process and shared by every data point. A text, in contrast, is a sequence of tokens whose length varies from data point to data point, and a graph varies in its node and edge sets. Such data points carry no attribute set of their own. One can be imposed, for example the number of occurrences of each word from a fixed word list, but such a set discards the order of the tokens. The table of weather measurements loses nothing in this way, since a day is fully described by its four attributes, while a text with its tokens in another order is a different text. Instead, feature learning methods are developed that map such data points to feature vectors (or embeddings) (Kipf and Welling, 2017; Mikolov et al., 2013).

In database theory, the relational model formalizes a table as a relation. A relation is a set of tuples of attribute values. The relational model also provides the operations, such as selecting rows or joining tables, that database systems use to manipulate tabular data (Silberschatz et al., 2019).

**See also:** [data](https://dictionaryofml.org/terms/data.html), [data point](https://dictionaryofml.org/terms/datapoint.html), [dataset](https://dictionaryofml.org/terms/dataset.html), [feature](https://dictionaryofml.org/terms/feature.html), [label](https://dictionaryofml.org/terms/label.html), [feature matrix](https://dictionaryofml.org/terms/featuremtx.html), relational model, missing data, [data imputation](https://dictionaryofml.org/terms/dataimputation.html), gradient-boosted decision tree, [feature learning](https://dictionaryofml.org/terms/featlearn.html).

## References

- Finnish Meteorological Institute (2024). The Finnish Meteorological Institute's open data. <https://opendata.fmi.fi/wfs>
- Everitt (2002). The Cambridge Dictionary of Statistics. *Cambridge Univ. Press*.
- Tennison et al. (2015). Model for Tabular Data and Metadata on the Web. <https://www.w3.org/TR/tabular-data-model/>
- Kipf and Welling (2017). Semi-Supervised Classification with Graph Convolutional Networks. *5th Int. Conf. Learn. Representations*. <https://openreview.net/forum?id=SJU4ayYgl>
- Mikolov et al. (2013). Efficient Estimation of Word Representations in Vector Space. *Int. Conf. Learn. Represent. (ICLR), Workshop Track*. <https://openreview.net/forum?id=idpCdOWtqXd60>
- Silberschatz et al. (2019). Database System Concepts. *McGraw-Hill Education*.

## Cite this entry

```bibtex
@misc{dictml_tabulardata,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {tabular data},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/tabulardata.html}
}
```


---

# trusted execution environment (TEE)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/tee.html) · [typeset PDF](https://dictionaryofml.org/terms/tee.pdf) · [LaTeX](https://dictionaryofml.org/terms/tee.tex) · [Python demo](https://dictionaryofml.org/terms/tee.py) · [notebook](https://dictionaryofml.org/terms/tee.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A trusted execution environment (TEE) is a part of a computing system that isolates code and data from other software. Even the operating system cannot directly read or alter its protected memory. Remote attestation can prove to another party which code it runs. Processor features such as Arm TrustZone and Intel SGX provide TEEs on ordinary chips. In machine learning, a TEE protects input feature vectors, learned model parameters and the integrity of the computed prediction. It does not limit what the prediction reveals; differential privacy addresses that problem. Physical and software side channels require additional countermeasures. Compared with fully homomorphic encryption, a TEE runs at near-native speed but replaces a mathematical guarantee with trust in the chip vendor and its hardware.

## Definition

Consider an embedded controller in a refrigerated medicine cabinet. Each minute, it reads a temperature sensor and forms a feature vector ${\mathbf x}$ from the current temperature and its recent changes. A learned hypothesis computes the prediction $\widehat{h}({\mathbf x})$ of whether the cooling system is failing. The controller sits in a sealed metal box with no network cable and no radio. Its only inputs are the sensor cable and the power cable, and its only output is an alarm display. The model parameters $\widehat{{\mathbf w}}$ of $\widehat{h}$ and the recorded temperature readings never leave the box.

A TEE is the part of the processor that provides the logical isolation. Software outside it cannot directly read or alter its protected code and memory, even if that software is a privileged operating system (Sabt et al., 2015). The metal enclosure is not the TEE; it is a separate layer that protects the processor against physical probing and fault injection.

Most TEEs are a mode of an ordinary processor. Arm TrustZone splits the processor of the controller into two parts. The normal world runs the display software. The secure world runs the prediction code and holds $\widehat{{\mathbf w}}$ and the temperature readings. The normal world cannot read the memory of the secure world even with root privileges (Pinto and Santos, 2019).

Intel SGX provides similar isolation for an enclave inside a process on a server (Costan and Devadas, 2016). Remote attestation lets a client verify a signed digest of the enclave's code before it sends any data. The isolation guarantee concerns direct access through software and memory interfaces; information revealed by outputs, side channels and physical attacks requires additional protection. Fig. 1 shows the embedded controller and what each part can see.

![Figure 1 of the entry tee](https://dictionaryofml.org/terms/fig/tee-fig1.svg)

*Figure 1: A TEE in an embedded temperature-monitoring controller. The secure world (the TEE) holds the prediction code, the model parameters and the recorded temperature readings; the normal world receives only the prediction for the alarm display. Separately, the sealed enclosure reduces the physical side channels available to a probe outside the processor*

A TEE protects three assets of a machine learning system. The first is the data: the temperature history is an operational record that the cabinet operator must not disclose or allow an attacker to alter. The second is the learned hypothesis. Its model parameters $\widehat{{\mathbf w}}$ are the outcome of training on a costly dataset. An attack that reads them out obtains the hypothesis without the dataset. The third is the integrity of the prediction: nobody can alter the code so that it suppresses an alarm when the cooling system is failing.

A TEE does not protect what the prediction reveals. A membership inference attack that only observes predictions is unaffected by it, and differential privacy addresses that part.

A TEE also does not, by itself, protect against physical side channels. The processor's current and electromagnetic field depend on the data it processes. A probe held near the chip can recover secret keys from these traces (Kocher et al., 1999). The same probe recovers the model parameters and the layer structure of an artificial neural network running on an embedded processor, one multiplication at a time (Batina et al., 2019). A short electromagnetic pulse or a glitch on the supply voltage can also skip an instruction, e.g., the check that rejects unsigned code (Bar-El et al., 2006).

The metal enclosure attenuates the emitted field and blocks the pulse. Tamper sensors erase $\widehat{{\mathbf w}}$ when the box is opened. Constant-time code makes the power trace independent of the data.

However, server TEEs are also exposed to software and processor-level side channels. The operating system can see the memory access pattern of an enclave. Therefore, machine learning code inside an enclave must be data-oblivious. Its access patterns must not depend on the data (Ohrimenko et al., 2016). Speculative execution has also leaked enclave memory outright (Van Bulck et al., 2018).

Two uses of TEEs beyond the sealed box are prediction in the cloud and federated learning. A hospital that rents a server can require an SGX enclave. It first uses attestation to verify the agreed prediction code. It then encrypts its feature vectors for the enclave, so the cloud provider never sees them (Ohrimenko et al., 2016).

In federated learning, a server enclave can provide a hardware-based implementation of secure aggregation. Clients encrypt their model parameters updates for the enclave. It decrypts and sums the updates, while the server operator sees only the sum. Separately, the TrustZone secure world of each phone can protect its local update before transmission; a system can use either protection or combine both (Mo et al., 2021).

A TEE, fully homomorphic encryption and differential privacy provide different guarantees by different means. Fully homomorphic encryption hides the data during computation without trusting the processor, at a cost of orders of magnitude in time and memory. A TEE obtains near-native speed by trusting the chip vendor, the hardware implementation and the countermeasures against side channels. Differential privacy bounds what the released prediction reveals about any single data point (Dwork and Roth, 2014, Def. 2.4) and can be combined with either.

**Synonyms:** secure enclave.

**See also:** fully homomorphic encryption, secure aggregation, differential privacy, privacy protection, privacy attack, [membership inference attack](https://dictionaryofml.org/terms/membershipinferenceattack.html), attack, federated learning, edge device, cloud computing, machine learning system, [trustworthy artificial intelligence](https://dictionaryofml.org/terms/trustAI.html).

## References

- Sabt et al. (2015). Trusted Execution Environment: What It is, and What It is Not. *Proc. IEEE Trustcom/BigDataSE/ISPA*. <https://doi.org/10.1109/Trustcom.2015.357>
- Pinto and Santos (2019). Demystifying Arm TrustZone: A Comprehensive Survey. *ACM Computing Surveys*. <https://doi.org/10.1145/3291047>
- Costan and Devadas (2016). Intel SGX Explained.
- Kocher et al. (1999). Differential Power Analysis. *Advances in Cryptology -- CRYPTO '99*. <https://doi.org/10.1007/3-540-48405-1_25>
- Batina et al. (2019). CSI NN: Reverse Engineering of Neural Network Architectures Through Electromagnetic Side Channel. *Proc. 28th USENIX Security Symposium*.
- Bar-El et al. (2006). The Sorcerer's Apprentice Guide to Fault Attacks. *Proceedings of the IEEE*. <https://doi.org/10.1109/JPROC.2005.862424>
- Ohrimenko et al. (2016). Oblivious Multi-Party Machine Learning on Trusted Processors. *Proc. 25th USENIX Security Symposium*.
- Van Bulck et al. (2018). Foreshadow: Extracting the Keys to the Intel SGX Kingdom with Transient Out-of-Order Execution. *Proc. 27th USENIX Security Symposium*.
- Mo et al. (2021). PPFL: Privacy-preserving Federated Learning with Trusted Execution Environments. *Proc. 19th Annual Int. Conf. Mobile Systems, Applications, and Services (MobiSys)*. <https://doi.org/10.1145/3458864.3466628>
- Dwork and Roth (2014). The Algorithmic Foundations of Differential Privacy. *Foundations and Trends® in Theoretical Computer Science*. <https://doi.org/10.1561/0400000042>

## Cite this entry

```bibtex
@misc{dictml_tee,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {trusted execution environment (TEE)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/tee.html}
}
```


---

# test set

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/testset.html) · [typeset PDF](https://dictionaryofml.org/terms/testset.pdf) · [LaTeX](https://dictionaryofml.org/terms/testset.tex) · [Python demo](https://dictionaryofml.org/terms/testset.py) · [notebook](https://dictionaryofml.org/terms/testset.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A test set contains data points used to evaluate a hypothesis delivered by a machine learning method. These points are used neither for training nor for model selection. A test set is therefore also called a hold-out set. Under the independent and identically distributed assumption, its average loss is an unbiased estimator of the risk of the learned hypothesis. A concentration inequality relates the accuracy of this estimate to the test set size. The estimate remains valid only if no design choice depends on the test set. Selection based on repeated evaluation converts it into a validation set; overlap with the training set is test set contamination.

## Definition

Consider a machine learning method that uses weather recordings from Krems an der Donau (Austria). It learns a hypothesis $\widehat{h}$ that predicts tomorrow's maximum temperature from today's maximum. A training set is used to learn a hypothesis via empirical risk minimization. A validation set is then used to compare hypotheses from different models, such as a linear model and polynomial regression. Both sets influence the selected hypothesis, so its loss on them is usually smaller than on data points that the method did not use (Hastie et al., 2009, Sect. 7.4; Shalev-Shwartz and Ben-David, 2014, Sect. 11.2.2). Assessing $\widehat{h}$ therefore requires data points from neither set.

A test set $\mathcal{D}^{(\rm test)}$ is a dataset of data points used to evaluate a hypothesis delivered by a machine learning method. Its data points enter neither the training of a model on the training set $\mathcal{D}^{(\rm train)}$ nor model selection based on a validation set $\mathcal{D}^{(\rm val)}$ (Fig. 1). A test set is therefore also called a hold-out set. Some texts instead use that name for the validation set (Shalev-Shwartz and Ben-David, 2014, Sect. 11.2). For the temperature forecaster, the test set consists of days absent from both other sets.

![Figure 1 of the entry testset](https://dictionaryofml.org/terms/fig/testset-fig1.svg)

*Figure 1: Three roles for data points. On the training set (filled circles), two hypotheses are learned via empirical risk minimization from different models: $\widehat{h}^{(1)}$ from a linear model (solid curve) and $\widehat{h}^{(2)}$ from degree-three polynomial regression (dashed curve). The validation set (open triangles) is used to select among them. The test set (open squares) enters neither training nor model selection*

Suppose that $\widehat{h}$ was learned and selected without using $\mathcal{D}^{(\rm test)}$. Its test error is the average loss

$$
\frac{1}{|\mathcal{D}^{(\rm test)}|} \sum_{{\mathbf z} \in \mathcal{D}^{(\rm test)}}
L\left({\mathbf z},\widehat{h} \right) \text{.}
$$

Here, $L\left({\mathbf z},\widehat{h} \right)$ is the loss, a number that quantifies the error on one data point. Under the independent and identically distributed assumption, the points in $\mathcal{D}^{(\rm train)}$, $\mathcal{D}^{(\rm val)}$, and $\mathcal{D}^{(\rm test)}$ are independent draws from the same probability distribution $p({\mathbf z})$. The test error is then an unbiased estimator of the risk of $\widehat{h}$ (Hastie et al., 2009). The risk is the expected loss on a data point drawn from $p({\mathbf z})$ independently of all three sets. The validation error is used to select $\widehat{h}$, so it tends to understate the risk of the selected hypothesis.

The sizes of the three sets are design choices. Under the independent and identically distributed assumption, the test error averages $|\mathcal{D}^{(\rm test)}|$ independent loss values. If the loss lies in $[0,1]$, a concentration inequality bounds the difference between test error and risk. For a confidence level $\delta \in (0,1)$, this difference is at most $\sqrt{\ln(2/\delta)/(2\,|\mathcal{D}^{(\rm test)}|)}$ with probability at least $1-\delta$ (Shalev-Shwartz and Ben-David, 2014, Thm. 11.1). Solving that bound for $|\mathcal{D}^{(\rm test)}|$ gives the test set size a desired difference and confidence level require.

The validation set needs a concentration inequality that covers every hypothesis compared during model selection at once, so its required size grows with their number (Shalev-Shwartz and Ben-David, 2014, Thm. 11.2). The training set receives the remaining data points. No fixed rule determines the proportions. Fig. 2 shows a common example: 50 percent for training and 25 percent each for validation and testing (Hastie et al., 2009, Sect. 7.2).

![Figure 2 of the entry testset](https://dictionaryofml.org/terms/fig/testset-fig2.svg)

*Figure 2: A dataset $\mathcal{D}$ split into the training set $\mathcal{D}^{(\rm train)}$, the validation set $\mathcal{D}^{(\rm val)}$, and the test set $\mathcal{D}^{(\rm test)}$, with the typical proportions of Hastie et al. (2009, Sect. 7.2)*

A test error remains a valid risk estimate only if the test set is used once, after all choices are fixed. Using $\mathcal{D}^{(\rm test)}$ to compare models converts it into a validation set, and its error into a validation error. Moving any of its data points into the training set is test set contamination (Goodfellow et al., 2016, Sect. 5.3).

In a machine learning competition, organizers compare models submitted by participants on a common test set (Hastie et al., 2009, Sect. 11.9). The organizers keep the test labels private and compute each error. Participants cannot use those labels to revise their models. The reported test error can therefore estimate risk, provided that the organizers limit repeated feedback on the same test set.

**Synonyms:** hold-out set.

**See also:** [training set](https://dictionaryofml.org/terms/trainset.html), [validation set](https://dictionaryofml.org/terms/valset.html), [model selection](https://dictionaryofml.org/terms/modelsel.html), risk, [generalization](https://dictionaryofml.org/terms/generalization.html), test set contamination, [data leakage](https://dictionaryofml.org/terms/dataleakage.html), [empirical risk minimization](https://dictionaryofml.org/terms/erm.html).

## References

- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Goodfellow et al. (2016). Deep Learning. *MIT Press*.

## Cite this entry

```bibtex
@misc{dictml_testset,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {test set},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/testset.html}
}
```


---

# training error

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-10-29. [HTML](https://dictionaryofml.org/terms/trainerr.html) · [typeset PDF](https://dictionaryofml.org/terms/trainerr.pdf) · [LaTeX](https://dictionaryofml.org/terms/trainerr.tex) · [Python demo](https://dictionaryofml.org/terms/trainerr.py) · [notebook](https://dictionaryofml.org/terms/trainerr.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

The training error of a hypothesis is its average loss over the training set, the empirical risk of the hypothesis on the data points used for training. For a hypothesis delivered by empirical risk minimization, it is the smallest empirical risk that any hypothesis of the hypothesis space attains. Because the same data points selected the hypothesis, the training error is often far below the risk, by a gap that grows with the size of the hypothesis space, so a hypothesis with zero training error can have a large risk. Under the independent and identically distributed assumption, the training error of every hypothesis is a noisy estimate of its risk, and empirical risk minimization compares all of them at once, so a generalization guarantee for the empirical risk minimization solution requires the deviation to be small uniformly over the hypothesis space. A training error far below the validation error indicates overfitting, and a large training error indicates underfitting.

## Definition

Twenty of the $1827$ days of five years, 2020 to 2024, at the weather station Krems an der Donau (Austria), every $92$nd day, are drawn in Fig. 1 as a scatterplot. Each day is a data point: its feature $x$ is the morning minimum temperature, its label $y$ the maximum temperature of the day. Three of the days, drawn in black, form a training set, the days used for training, for two empirical risk minimization-based methods that deliver the hypotheses $\widehat{h}^{(1)}$ and $\widehat{h}^{(2)}$. The hypothesis $\widehat{h}^{(1)}$ is learned by empirical risk minimization with the squared error loss over the hypothesis space of affine maps $h(x) = b + w x$, the linear model. The dotted bars are its prediction errors on the three days, the differences between the recorded maximum and the value of $\widehat{h}^{(1)}$, and the average of the three squared bars is its training error, $7.91$.

The second hypothesis $\widehat{h}^{(2)}$ is learned by empirical risk minimization over the hypothesis space of all polynomials of degree at most two. Although the training error of $\widehat{h}^{(2)}$ is zero, and therefore far below that of $\widehat{h}^{(1)}$, it is the worse choice: averaged over all days of the five years, the loss of $\widehat{h}^{(2)}$ is $42.90$ against $24.22$ for $\widehat{h}^{(1)}$.

![Figure 1 of the entry trainerr](https://dictionaryofml.org/terms/fig/trainerr-fig1.svg)

*Figure 1: Twenty of the $1827$ days at Krems an der Donau, 2020 to 2024 (every $92$nd day), each a gray dot with the morning minimum temperature as its feature and the maximum temperature of the day as its label. The three black dots form the training set. The solid line is the hypothesis $\widehat{h}^{(1)}$ that empirical risk minimization learns from the three days over the linear model, and the dotted bars are its prediction errors on them; the average of the three squared bars is its training error. The dashed curve is the degree-two polynomial $\widehat{h}^{(2)}$ that empirical risk minimization learns over the polynomials of degree at most two, whose training error is zero; left of the coldest training day it falls below the frame. Data generated by pythondemos/trainerr.py*

Holding the intercept of $\widehat{h}^{(1)}$ at the value empirical risk minimization gave it and varying only its slope $w$ leaves the training error a function of one number. Fig. 2 shows it with the risk, the expected loss over the probability distribution of a day, approximated by the average over the five years. The two functions have different minimizers: the training error is smallest at the slope that empirical risk minimization picked, the risk at a smaller slope. At the slope empirical risk minimization picked, the training error is a third of the risk. The training error is an average over the three days that determined the hypothesis; it does not measure the loss on a day outside the training set.

![Figure 2 of the entry trainerr](https://dictionaryofml.org/terms/fig/trainerr-fig2.svg)

*Figure 2: The training error (solid curve) and the risk (dashed curve) of the hypothesis $h^{(w)}(x) = 10.21 + w x$ as functions of its slope $w$, for the three training days and the five years of Fig. 1; the intercept is the one empirical risk minimization gave $\widehat{h}^{(1)}$. The training error is smallest at the slope that empirical risk minimization picks (filled circle), where it is far below the risk; the risk is smallest at a smaller slope (open square). Data generated by pythondemos/trainerr.py*

The training error $E_{t}$ of a hypothesis $\widehat{h}$ is its average loss over the training set $\mathcal{D}^{(\rm train)} = \{{\mathbf z}^{(r)}\}_{r=1}^{m}$,

$$
E_{t} = \frac{1}{m} \sum_{r=1}^{m}
L\left({\mathbf z}^{(r)},\widehat{h} \right) \text{,}
$$

i.e., the empirical risk of $\widehat{h}$ on the data points used for learning $\widehat{h}$. The hypothesis is the output of a training method, a map $\mathcal{A}$ that reads in a training set and delivers $\widehat{h} = \mathcal{A}(\mathcal{D}^{(\rm train)})$. For a hypothesis delivered by empirical risk minimization, the training error is the smallest empirical risk that any hypothesis of the hypothesis space attains on $\mathcal{D}^{(\rm train)}$, since empirical risk minimization minimizes this average.

The training error is computed on the same data points that selected $\widehat{h}$, which limits its use as a measure of quality. The goal of a machine learning method is to learn a hypothesis that incurs a small loss on any data point, also those outside the training set, that is, to minimize the risk introduced above. The training error of the learned hypothesis is often a poor estimator for its risk, because the learned hypothesis has been adapted to the training set by minimizing the training error. A hypothesis with zero training error can therefore have a large risk, as $\widehat{h}^{(2)}$ of Fig. 1 does. The difference between the risk and the training error is the generalization gap.

Under the independent and identically distributed assumption, the training error becomes a random variable, as it depends on the randomly generated training set. For one fixed hypothesis $h \in \mathcal{H}$, the empirical risk $\widehat{L}\big(h|\mathcal{D}^{(\rm train)}\big)$ is an average of the $m$ loss values $L\left({\mathbf z}^{(1)},h \right), \ldots,
L\left({\mathbf z}^{(m)},h \right)$, which are realizations of independent and identically distributed RVs. It is therefore the realization of a random variable with expectation $\bar{L} \big( h \big)$ and a variance proportional to $1/m$. Empirical risk minimization does not evaluate this estimate at one fixed hypothesis: it compares the estimates of all hypotheses in $\mathcal{H}$ and picks the smallest. The estimates form the family $\{ \widehat{L}\big(h|\mathcal{D}^{(\rm train)}\big) \}_{h \in \mathcal{H}}$, a stochastic process indexed by the hypothesis space: one random variable for each hypothesis, all realized by the single draw of $\mathcal{D}^{(\rm train)}$. The minimizer $\widehat{h}$ of this process is itself a random variable, and its training error $E_{t} = \widehat{L}\big(\widehat{h}|\mathcal{D}^{(\rm train)}\big)$ is the minimum of many noisy estimates, so it is below $\bar{L} \big( \widehat{h} \big)$ on average.

A guarantee of generalization for the empirical risk minimization solution must therefore hold uniformly over the hypothesis space: if

\begin{equation}
\tag{1}
\sup_{h \in \mathcal{H}}
\big| \widehat{L}\big(h|\mathcal{D}^{(\rm train)}\big) - \bar{L} \big( h \big) \big|
\leq \epsilon \text{,}
\end{equation}

then $\bar{L} \big( \widehat{h} \big) \leq E_{t} + \epsilon$, and the risk of $\widehat{h}$ exceeds the smallest risk in $\mathcal{H}$ by at most $2\epsilon$ (Shalev-Shwartz and Ben-David, 2014, Lemma 4.2). A hypothesis space for which the uniform deviation Eq. (1) falls below any $\epsilon$ with probability at least $1-\delta$ once $m$ is large enough has the uniform convergence property (Shalev-Shwartz and Ben-David, 2014, Def. 4.3). For a finite hypothesis space and a loss with values in $[0,1]$, a union bound over the hypotheses and Hoeffding's inequality give $\epsilon = \sqrt{\ln(2 |\mathcal{H}| / \delta) / (2 m)}$ (Shalev-Shwartz and Ben-David, 2014, Cor. 4.6). For an infinite hypothesis space, $\ln |\mathcal{H}|$ is replaced by a measure of its size such as the Vapnik–Chervonenkis dimension or the Rademacher complexity (Shalev-Shwartz and Ben-David, 2014, Thm. 6.8, Thm. 26.5).

In contrast, the validation error, computed as the average loss on data points not used in the training, is often a much better estimator for the risk of the learned hypothesis. For example, let the data points of the validation set be realizations of independent and identically distributed RVs, independent of the training set. Each summand of the validation error is then the loss on an independent draw from the probability distribution of a data point, whose expectation conditional on the training set is the risk of the learned hypothesis (Hastie et al., 2009, Eq. (7.2)). The expectation of the validation error conditional on the training set is therefore that risk. Moreover, the validation error estimates the risk of the single hypothesis that $\mathcal{A}$ delivered, so one Hoeffding bound without $|\mathcal{H}|$ suffices for it, and it concentrates around the risk at a rate that depends only on the size of the validation set, not on how the hypothesis was learned (Shalev-Shwartz and Ben-David, 2014, Thm. 11.1).

Fig. 3 shows three deviations of a training error from a risk for one year of the Krems record: for one fixed hypothesis, for the hypothesis that empirical risk minimization learns, and the largest deviation over the hypothesis space. The $366$ days of the year stand in for the probability distribution of a day, so the risk of a hypothesis is its average loss over the year, and a training set of $m$ days is drawn at random from the year, $300$ times for each $m \in \{10, 20, 40, 80\}$. The hypothesis space is the linear model, which consists of all hypothesis maps $h(x) = w_{1} x + w_{0}$ with $w_{1}, w_{0} \in \mathbb{R}$.

The fixed hypothesis is the one with the smallest risk in the hypothesis space, the hypothesis learned by linear regression from all $366$ days; it does not depend on the draw. The empirical risk minimization hypothesis is learned by linear regression from the drawn training set. The largest deviation over the hypothesis space, the uniform deviation Eq. (1), is approximated by the maximum over a grid of $2500$ hypotheses around the learned one. Over the repeated draws, the deviation of the fixed hypothesis has mean zero. The training error of the empirical risk minimization hypothesis is below its risk on average, and the largest deviation is several times the standard deviation of the fixed hypothesis' deviation. The gap of the empirical risk minimization hypothesis never exceeds the largest deviation, as the uniform deviation Eq. (1) requires, and both decrease as the training set grows (pythondemos/trainerr.py).

![Figure 3 of the entry trainerr](https://dictionaryofml.org/terms/fig/trainerr-fig3.svg)

*Figure 3: Deviation of the training error from the risk over repeated random draws of a training set of fixed size from the days of 2024 at Krems an der Donau, mean and one standard deviation either side. For one fixed hypothesis (filled circles) the deviation has mean zero. For the hypothesis that empirical risk minimization learns from the draw (filled squares) it is negative on average. The largest deviation over the hypothesis space (open triangles), the uniform deviation Eq. (1) approximated on a grid of $2500$ hypotheses, bounds the empirical risk minimization gap in every draw and decreases with the size of the training set. Data generated by pythondemos/trainerr.py*

The size of the gap $\bar{L} \big( \widehat{h} \big) - E_{t}$ depends on the hypothesis space. The first day of each month from January to August 2024 forms a training set of eight days, and the rest of the year is held back. A hypothesis of the linear model learned from the eight days has a training error of $20.36$, only slightly below its average squared error of $21.60$ on the held-back days: the linear model has two model parameters and can adapt little to the particular eight days. Fig. 4 shows the two errors for polynomials of prescribed maximum degree $p = 0, \ldots, 7$. Increasing the degree means a larger hypothesis space, a polynomial of degree $p$ being one of degree $p+1$ with a zero leading coefficient, so the training error never increases with the degree. In contrast, the average loss on the held-back days is smallest at degree one and grows for larger degrees, where the polynomial oscillates between the eight days it was learned from.

![Figure 4 of the entry trainerr](https://dictionaryofml.org/terms/fig/trainerr-fig4.svg)

*Figure 4: Training error (solid line, filled circles) and held-back error (dashed line, filled squares) of a polynomial $\widehat{h}(x)$ of prescribed maximum degree learned by empirical risk minimization from eight days of 2024 at Krems an der Donau. The training error never increases with the degree, since increasing the degree means a larger hypothesis space. In contrast, the error on the days outside the training set is smallest at a low degree and grows for larger degrees. Data generated by pythondemos/trainerr.py*

By comparing the training error of a machine learning method with the resulting validation error, two failures can be detected. A training error far below the validation error indicates overfitting (Bishop, 2006, Sect. 1.1). A large training error indicates underfitting: a hypothesis space too small, or an iterative optimizer that stopped too early, which restricts the effective hypothesis space. Fig. 4 also shows the effect of regularization by model pruning: restricting the hypothesis space by using a small polynomial degree raises the training error a little and lowers the average loss on the held-back days.

**See also:** [training set](https://dictionaryofml.org/terms/trainset.html), [validation set](https://dictionaryofml.org/terms/valset.html), empirical risk, [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), [loss](https://dictionaryofml.org/terms/loss.html), [validation error](https://dictionaryofml.org/terms/valerr.html), risk, generalization gap, [overfitting](https://dictionaryofml.org/terms/overfitting.html), underfitting, [hypothesis](https://dictionaryofml.org/terms/hypothesis.html), [independent and identically distributed assumption](https://dictionaryofml.org/terms/iidasspt.html), [generalization](https://dictionaryofml.org/terms/generalization.html), stochastic process, concentration inequality, Vapnik–Chervonenkis dimension.

## References

- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>

## Cite this entry

```bibtex
@misc{dictml_trainerr,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {training error},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/trainerr.html}
}
```


---

# training

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-10. [HTML](https://dictionaryofml.org/terms/training.html) · [typeset PDF](https://dictionaryofml.org/terms/training.pdf) · [LaTeX](https://dictionaryofml.org/terms/training.tex) · [Python demo](https://dictionaryofml.org/terms/training.py) · [notebook](https://dictionaryofml.org/terms/training.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Training is the process of choosing a useful hypothesis out of a hypothesis space, or model. Mathematically, training is an optimization problem with an objective function that a training method builds from the data points it reads. In supervised learning the objective function is the average loss over data points whose label values are known, such as the average squared error loss. In unsupervised learning it measures structure instead, such as the clustering error incurred by a choice of cluster centroids. In reinforcement learning it is the expected cumulative reward collected by a policy. Training methods differ also in how they access data points. Offline methods require the training set up front, while online methods learn a hypothesis sequentially, updating it whenever new data points arrive.

## Definition

A shop forecasts daily bread demand from the weather with a linear model. It trains that linear model by gradient descent on a year of past records. To let the forecaster follow a changing clientele instead, it trains the same linear model by online gradient descent on each new day's sales: the same model and the same objective function, with a different form of data access. Training is the process by which a machine learning method picks one hypothesis out of a hypothesis space. The hypothesis space, or model, is the set of candidate maps the method may choose from. Training selects the single hypothesis $\widehat{h}$ that the method then uses for inference: the training phase learns $\widehat{h}$, the inference phase uses it to deliver predictions for newly arriving data points. What makes one candidate better than another is an objective function, so training amounts to solving an optimization problem.

Training methods differ in two respects. The first is the objective function. In supervised learning each data point carries a label, and the objective function is the average loss of a hypothesis over the training set, its empirical risk, such as the average squared error loss in linear regression. In unsupervised learning no label is given, so the objective function measures a structural quantity instead, such as the clustering error incurred by a choice of cluster centroids in hard clustering. In reinforcement learning there is no fixed training set at all: the objective function is the expected cumulative reward $G$ collected by a policy that interacts with an environment.

The second respect is the form of data access, offline or online. A training method is an algorithm $\mathcal{A}$ that reads in a dataset $\mathcal{D}$ of data points, constructs the objective function from them together with the chosen model and loss, and delivers the hypothesis making that objective function small, $\widehat{h} = \mathcal{A}(\mathcal{D})$. An offline algorithm reads in all data points at once, computes, and delivers $\widehat{h}$. An online algorithm reads in new data points as they arrive, updating the current hypothesis at each time instant $t = 1, 2, \ldots$ instead of waiting for a complete training set. The hypotheses it produces form a Markov chain, since each one depends on the dataset only through its predecessor and the data point that just arrived (Fig. 1).

![Figure 1 of the entry training](https://dictionaryofml.org/terms/fig/training-fig1.svg)

*Figure 1: Two forms of data access. Left: an offline algorithm $\mathcal{A}$ reads in the entire dataset $\mathcal{D}$ and delivers a learned hypothesis $\widehat{h}$. Right: an online algorithm holds a current hypothesis $\widehat{h}^{(t)}$; the data point ${\mathbf z}^{(t)}$ arriving at time instant $t$ moves it to $\widehat{h}^{(t+1)}$, so the hypotheses form a Markov chain: the next one depends on the dataset only through the current one and the arriving data point*

For a parametric model, each hypothesis $h^{({\mathbf w})}$ is fixed by a choice of model parameters ${\mathbf w}$, so training searches over ${\mathbf w}$. Many training methods amount to iterating an operator on those model parameters,

$$
{\mathbf w}^{(t+1)} = T({\mathbf w}^{(t)})
\text{, for } t = 0,1,\ldots \text{,}
$$

and the two forms of data access appear here as two forms of the operator. An offline method builds its objective function once, from the whole dataset, so $T$ is fixed: the same map is applied at every step, and training stops at, or near, a fixed point of $T$, where a further iteration leaves the model parameters unchanged. An online method has no fixed objective function. Its operator reads in the arriving data point,

$$
{\mathbf w}^{(t+1)} = T\big({\mathbf w}^{(t)},
{\mathbf z}^{(t)}\big) \text{,}
$$

so the map changes from step to step, and in general there is no single fixed point to stop at. The model parameters keep changing for as long as new data points arrive. The learned hypothesis $\widehat{h}$ is the one that the final model parameters define.

What $T$ does is decided by the shape of the objective function. When the objective function is differentiable in all model parameters at once, $T$ is a gradient step, $T({\mathbf w}) = {\mathbf w} - \eta \nabla f({\mathbf w})$ with step size $\eta$ (Boyd and Vandenberghe, 2004, Sect. 9.3). The familiar methods are then one operator applied to three different objective functions, which is the second respect, data access, again. Offline empirical risk minimization takes the gradient of the average loss $f({\mathbf w})$ over the whole training set. When that training set is too large to pass through at every step, stochastic gradient descent replaces $\nabla f({\mathbf w})$ by the gradient of the average loss over a few data points (Bottou, 2010). Its operator is drawn at random at each step, from a distribution that the training set fixes, so it is fixed in expectation rather than fixed outright. In online learning there is no training set at all, and online gradient descent takes one gradient step on the loss of the arriving data point alone (Zinkevich, 2003).

Some objective functions arising in training methods can be minimized easily once a subset of the model parameters is held fixed. Then $T$ is one sweep of alternating optimization, a minimization over each subset in turn (Beck, 2017, Ch. 14). The k-means algorithm alternates between assigning each data point to the nearest cluster and recomputing the cluster means (Lloyd, 1982). The expectation–maximization algorithm is the probabilistic instance: training a Gaussian mixture model, it alternates between the responsibilities of the mixture components for each data point and the parameters of those components (Dempster et al., 1977). What separates expectation–maximization from a plain alternation is that each step minimizes a surrogate, a bound on the objective function rather than the objective function itself, which is why the likelihood function cannot decrease (Bishop, 2006, Sect. 9.4). Both are offline. The k-means algorithm reaches a fixed point of its operator after finitely many steps, since every step lowers the objective function and there are finitely many assignments (Bishop, 2006, Sect. 9.1); for expectation–maximization, under a mild continuity condition, the likelihood function converges and every limit point of the iterates is a stationary point of it (Wu, 1983, Thm. 2). Neither is guaranteed to reach the smallest loss available in $\mathcal{H}$: $k$-means can stop at a local minimum of its objective function (Bishop, 2006, Sect. 9.1), and a stationary point of the likelihood function need not maximize it.

Every operator above makes the objective function small on the data points it is given. One of the main challenges in machine learning is to control the discrepancy between the loss on the training set and the loss on data points outside it, and no choice of optimization method removes that challenge (Goodfellow et al., 2016).

**Synonyms:** model fitting, fitting.

**See also:** optimization problem, inference, [supervised learning](https://dictionaryofml.org/terms/supervisedlearning.html), [unsupervised learning](https://dictionaryofml.org/terms/unsupervisedlearning.html), [reinforcement learning](https://dictionaryofml.org/terms/reinforcementlearning.html), [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), [gradient descent](https://dictionaryofml.org/terms/gd.html), [stochastic gradient descent](https://dictionaryofml.org/terms/stochGD.html), [online learning](https://dictionaryofml.org/terms/onlinelearning.html), [expectation–maximization](https://dictionaryofml.org/terms/em.html), [Gaussian mixture model](https://dictionaryofml.org/terms/gmm.html), [$k$-means](https://dictionaryofml.org/terms/kmeans.html), fixed point, [model](https://dictionaryofml.org/terms/model.html), model parameter, [loss](https://dictionaryofml.org/terms/loss.html).

## References

- Boyd and Vandenberghe (2004). Convex Optimization. *Cambridge Univ. Press*. <https://doi.org/10.1017/CBO9780511804441>
- Bottou (2010). Large-scale machine learning with stochastic gradient descent. *Proceedings of COMPSTAT'2010*. <https://doi.org/10.1007/978-3-7908-2604-3_16>
- Zinkevich (2003). Online Convex Programming and Generalized Infinitesimal Gradient Ascent. *Proc. 20th Int. Conf. Mach. Learn.*.
- Beck (2017). First-Order Methods in Optimization. *SIAM-Society for Industrial and Applied Mathematics*.
- Lloyd (1982). Least squares quantization in PCM. *IEEE Trans. Inf. Theory*. <https://doi.org/10.1109/TIT.1982.1056489>
- Dempster et al. (1977). Maximum likelihood from incomplete data via the EM algorithm. *J. Roy. Statist. Soc.: Ser. B (Methodological)*. <https://doi.org/10.1111/j.2517-6161.1977.tb01600.x>
- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Wu (1983). On the convergence properties of the EM algorithm. *Ann. Statist.*. <https://doi.org/10.1214/aos/1176346060>
- Goodfellow et al. (2016). Deep Learning. *MIT Press*.

## Cite this entry

```bibtex
@misc{dictml_training,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {training},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/training.html}
}
```


---

# training set

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-02. [HTML](https://dictionaryofml.org/terms/trainset.html) · [typeset PDF](https://dictionaryofml.org/terms/trainset.pdf) · [LaTeX](https://dictionaryofml.org/terms/trainset.tex) · [Python demo](https://dictionaryofml.org/terms/trainset.py) · [notebook](https://dictionaryofml.org/terms/trainset.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A training set is a dataset used to train a model: it consists of data points for which the loss incurred by each candidate hypothesis can be evaluated, so that training methods can pick the hypothesis that performs best on it. Empirical risk minimization, for example, learns the hypothesis minimizing the average loss on the training set, the empirical risk, whose minimum value is the training error. Because the learned hypothesis is picked to perform well on the training set, a small training error can be misleading and result in overfitting. The validation error on a held-back validation set indicates whether the learned hypothesis also predicts well outside the training set.

## Definition

Consider three days of weather recordings: for each day, the morning minimum temperature is the feature and the maximum daytime temperature is the label. The ultimate goal of a machine learning method is to learn a hypothesis (or to train a model) that incurs a small prediction error not only on the data points used for training but on any data point that is typical for the application. The prediction error is measured by some loss function: the loss is a number, typically nonnegative, that quantifies the error of a hypothesis on a single data point, such as the squared difference between the predicted and the true label. The goal is formalized probabilistically as a small risk: the expected loss under the probability distribution from which the data points are drawn (independent and identically distributed assumption). The method must therefore compare the candidate hypotheses from its hypothesis space on data points for which the loss can be evaluated. For example, in regression or classification, the loss can only be computed for data points with a known label. These data points form a dataset referred to as the training set, $\mathcal{D}^{(\rm train)} = \{{\mathbf z}^{(r)}\}_{r=1}^{m}$.

A training method is a map $\mathcal{A}$ whose input is the training set and whose output is a learned hypothesis, $\widehat{h} = \mathcal{A}(\mathcal{D}^{(\rm train)})$. Training evaluates this map on the given training set. For example, empirical risk minimization minimizes the average loss on the training set, the empirical risk, to learn a hypothesis

$$
\widehat{h} \in \mathop{\mathrm{arg\,min}}_{h \in \mathcal{H}}
\frac{1}{m}
\sum_{r=1}^{m}
L\left({\mathbf z}^{(r)},h \right) \text{.}
$$

The empirical risk of the learned hypothesis is the training error.

Fig. 1 shows a training set of the three days along with two hypotheses learned from it: the linear hypothesis $\widehat{h}^{(1)}$ delivered by empirical risk minimization on the linear model, and a degree-two polynomial $\widehat{h}^{(2)}$ that passes through all three data points and therefore attains a training error of zero.

![Figure 1 of the entry trainset](https://dictionaryofml.org/terms/fig/trainset-fig1.svg)

*Figure 1: Data points representing daily weather conditions: each data point is a day, with the morning minimum temperature as its feature and the maximum daytime temperature as its label. The three filled circles form the training set $\mathcal{D}^{(\rm train)}$. Both curves are hypotheses learned from $\mathcal{D}^{(\rm train)}$: the linear hypothesis $\widehat{h}^{(1)}$ (solid curve) minimizes the average squared error on $\mathcal{D}^{(\rm train)}$ over the linear model, while the dashed degree-two polynomial $\widehat{h}^{(2)}$ passes through all three data points and attains a training error of zero. Both incur a small average loss on $\mathcal{D}^{(\rm train)}$, but between and beyond the three circles the two curves are far apart: at the largest morning minimum temperature shown, the two predicted maximum temperatures differ by about seven degrees. Data generated by pythondemos/trainset.py*

Since $\widehat{h}$ is chosen to make the empirical risk small, the training error is typically smaller than the loss that the same hypothesis incurs on data points outside $\mathcal{D}^{(\rm train)}$ (Hastie et al., 2009, Sect. 7.4). A small training error therefore does not guarantee a small loss outside the training set. Fig. 1 illustrates this: the polynomial $\widehat{h}^{(2)}$ attains a training error of zero, the linear hypothesis $\widehat{h}^{(1)}$ does not. The figure shows no data points outside $\mathcal{D}^{(\rm train)}$, but it shows that the two curves predict different labels for feature values between and beyond the three training data points. At most one of the two curves can therefore incur a small loss on a new data point with such a feature value, and the training error cannot tell which one (see overfitting).

A second subset of data points, the validation set $\mathcal{D}^{(\rm val)}$, is therefore held back from model training (Fig. 2): the average loss on $\mathcal{D}^{(\rm val)}$, the validation error, evaluates the learned hypothesis on data points it was not trained on. Comparing the two errors diagnoses the machine learning method: if both are small, the learned hypothesis is likely to predict well outside $\mathcal{D}^{(\rm train)}$ as well. A small training error paired with a validation error far above it indicates overfitting; one typical cause is a hypothesis space that is too large for the number of data points in $\mathcal{D}^{(\rm train)}$ (Bishop, 2006, Sect. 1.1; Hastie et al., 2009, Sect. 7.2).

![Figure 2 of the entry trainset](https://dictionaryofml.org/terms/fig/trainset-fig2.svg)

*Figure 2: The available dataset split into two disjoint subsets. Only the shaded subset, the training set $\mathcal{D}^{(\rm train)}$, enters the training method $\mathcal{A}$ that delivers a learned hypothesis $\widehat{h}$; the validation set is what remains to estimate how that hypothesis behaves on data points it was not trained on*

Which data points are contained in $\mathcal{D}^{(\rm train)}$ decides what is learned. The choice of the training set therefore matters for trustworthy artificial intelligence: a training set that underrepresents a group of data points can yield a learned hypothesis that incurs a larger loss on that group, a source of unfairness (see fairness) (Barocas et al., 2023; Buolamwini and Gebru, 2018; Mehrabi et al., 2021).

**See also:** [training](https://dictionaryofml.org/terms/training.html), [dataset](https://dictionaryofml.org/terms/dataset.html), [validation set](https://dictionaryofml.org/terms/valset.html), [test set](https://dictionaryofml.org/terms/testset.html), [data point](https://dictionaryofml.org/terms/datapoint.html), [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), [hypothesis](https://dictionaryofml.org/terms/hypothesis.html), [loss](https://dictionaryofml.org/terms/loss.html), [training error](https://dictionaryofml.org/terms/trainerr.html), [validation error](https://dictionaryofml.org/terms/valerr.html), [hypothesis space](https://dictionaryofml.org/terms/hypospace.html), [overfitting](https://dictionaryofml.org/terms/overfitting.html).

## References

- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Barocas et al. (2023). Fairness and Machine Learning: Limitations and Opportunities. *MIT Press*. <https://fairmlbook.org/>
- Buolamwini and Gebru (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. *Proceedings of the 1st Conference on Fairness, Accountability and Transparency*. <https://proceedings.mlr.press/v81/buolamwini18a.html>
- Mehrabi et al. (2021). A Survey on Bias and Fairness in Machine Learning. *ACM Computing Surveys*. <https://doi.org/10.1145/3457607>

## Cite this entry

```bibtex
@misc{dictml_trainset,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {training set},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/trainset.html}
}
```


---

# transfer learning

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-07. [HTML](https://dictionaryofml.org/terms/transferlearning.html) · [typeset PDF](https://dictionaryofml.org/terms/transferlearning.pdf) · [LaTeX](https://dictionaryofml.org/terms/transferlearning.tex) · [Python demo](https://dictionaryofml.org/terms/transferlearning.py) · [notebook](https://dictionaryofml.org/terms/transferlearning.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Transfer learning is the reuse of what was learned on one learning task to solve another learning task. A learning task can be characterized by a probability distribution of data points, and the effectiveness of transfer learning depends on the similarity of the corresponding probability distributions. Task similarity can be encoded as a similarity graph whose edge weights couple the learning tasks at its nodes; transfer learning methods can then be obtained by optimization over this graph.

## Definition

The weather stations Krems an der Donau and Melk in the Wachau valley each record the minimum and the maximum temperature of every day. Predicting the maximum temperature $y$ of a day at one station, the label, from its minimum temperature $x$, the feature, is a learning task. Each weather station defines a separate learning task. Fig. 1 shows the two tasks side by side, each with a decision tree of depth two, the hypothesis learned from a training set of five labeled days of 2020. A learning task can be characterized by the probability distribution of its data points: at the two stations those distributions are similar. Transfer learning uses such similarity. It couples the learning tasks so that what is learned at one station improves the predictions at the other, and its effectiveness depends on how similar the probability distributions of the coupled tasks are (Pan and Yang, 2010).

![Figure 1 of the entry transferlearning](https://dictionaryofml.org/terms/fig/transferlearning-fig1.svg)

*Figure 1: Two learning tasks in the Wachau valley. (a) The course of the Danube and four weather stations around the valley. (b) and (c) The days of 2020 at Krems an der Donau and at Melk, with the minimum temperature of a day as feature $x$ and its maximum temperature as label $y$. At each station, five labeled days (filled circles) are the training set. The dotted step function is the decision tree learned from those five days alone; the solid one is the decision tree learned when the two tasks are coupled by requiring their predictions to agree on the test features $\mathcal{T}$ ($20$ of the $40$, every second, as ticks along the horizontal axis). Data generated by pythondemos/transferlearning.py*

The coupling in Fig. 1 is implemented by adding a penalty to the objective function of the model training at each station. Let $h^{(1)}$ and $h^{(2)}$ be the hypotheses of the two stations and $\mathcal{T}$ a test set of feature values selected at random from the range $[-10, 10]$ of minimum temperatures. Transfer learning then minimizes the sum of the two training errors plus $\alpha \frac{1}{|\mathcal{T}|} \sum_{x \in \mathcal{T}}
\big( h^{(1)}(x) - h^{(2)}(x)
\big)^{2}$, the mean squared disagreement of the two hypotheses on the test features. For decision trees, minimizing over one tree while the other is held fixed is a weighted training of that tree on its own days together with the test features labeled by the other tree. Alternating the two minimizations is a block coordinate descent for the coupled problem: an exact minimization over one tree cannot increase the coupled objective function, so the sequence of objective values is nonincreasing. A decision tree is learned greedily, so each step is only an approximate block minimization, and the alternation is a heuristic for the coupled problem rather than a method that solves it. With five labeled days per station, the coupled trees agree on the test features (mean squared disagreement $32.2 \to 0.1$), and their average error on the held-out days of the two stations falls from $47.4$ to $33.5$.

The coupling of two tasks extends to a collection of learning tasks, one per node $i = 1, \ldots,
n$ of an undirected weighted graph $\mathcal{G}$, the similarity graph. A weighted edge $\{i,i'\}$ with edge weight $A_{i,i'}$ encodes the similarity between two learning tasks; a zero weight leaves them uncoupled. Each node $i$ carries a local model, a local dataset, and the local hypothesis $h^{(i)}$ learned from them. Minimizing the sum of the local training errors plus $\alpha$ times the generalized total variation

$$
\sum_{\{i,i'\} \in \mathcal{E}}
A_{i,i'}
d^{(i,i')} \text{,}
$$

with $d^{(i,i')}$ a measure of the discrepancy between the local hypotheses $h^{(i)}$ and $h^{(i')}$, is generalized total variation minimization (SarcheshmehPour et al., 2023). The coupling in Fig. 1 is generalized total variation minimization on a graph with two nodes, with the mean squared disagreement on the test features $\mathcal{T}$ as the discrepancy measure; this measure applies to any local model, the decision trees included. For local models with model parameters $\mathbf{w}^{(i)}$, the squared distance $\left\lVert \mathbf{w}^{(i)} -
\mathbf{w}^{(i')} \right\rVert_{2}^{2}$ is a common choice, and the network Lasso is the variant with the unsquared distance (Hallac et al., 2015). Generalized total variation minimization learns all local hypotheses simultaneously and transfers knowledge between well-connected learning tasks, as in federated learning. Fig. 2 shows a similarity graph of nine stations in Lower Austria, with edge weights that fall off with the distance between stations and are kept for the nearest neighbors of each. With five labeled days and a decision tree at every station, generalized total variation minimization with the disagreement on $\mathcal{T}$ lowers the average held-out error of the nine stations from $57.0$ to $35.9$ and the error at eight of them; at the ninth, Wien, whose edges are the weakest of the graph, the coupling raises it slightly.

![Figure 2 of the entry transferlearning](https://dictionaryofml.org/terms/fig/transferlearning-fig2.svg)

*Figure 2: A similarity graph of nine weather stations in Lower Austria, each a node carrying its own learning task. The edge weight between two stations is $\exp(-d^{2}/(2 s^{2}))$ for their distance $d$ and $s = 40$ kilometers, kept for the three nearest neighbors of every station: thick edges weigh at least $0.5$, thin edges less. Data generated by pythondemos/transferlearning.py*

A special case of transfer learning is a node without any labeled data points. Its own training error is absent from the objective, so generalized total variation minimization determines its hypothesis through the generalized total variation alone, from the predictions of its neighbors on the test features. This is semi-supervised learning: the unlabeled node still has data points, and its position in the similarity graph is what lets them be used (Anon., 2006).

A second form of transfer learning places the similarity between a pretraining task and the new one, instead of between the stations of a similarity graph. It adapts a deep net trained on a large dataset to a small one: the features computed by its early layers are kept (see feature learning) and the remaining model parameters are adapted by fine-tuning. When the features useful for the first task are not useful for the second, the adapted hypothesis can have a larger risk than one trained from scratch, which is called negative transfer (Pan and Yang, 2010). The case in which the learning task stays fixed and only the probability distribution of the feature vectors changes is domain adaptation (Pan and Yang, 2010, Sect. 2.3).

**See also:** learning task, generalized total variation minimization, generalized total variation, local model, local dataset, edge weight, semi-supervised learning, multitask learning, [fine-tuning](https://dictionaryofml.org/terms/finetuning.html), domain adaptation, federated learning.

## References

- Pan and Yang (2010). A Survey on Transfer Learning. *IEEE Transactions on Knowledge and Data Engineering*. <https://doi.org/10.1109/TKDE.2009.191>
- SarcheshmehPour et al. (2023). Clustered Federated Learning via Generalized Total Variation Minimization. *IEEE Trans. Signal Process.*. <https://doi.org/10.1109/TSP.2023.3322848>
- Hallac et al. (2015). Network Lasso: Clustering and Optimization in Large Graphs. *Proc. 21st ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining*. <https://doi.org/10.1145/2783258.2783313>
- Anon. (2006). Semi-Supervised Learning. *MIT Press*. <https://doi.org/10.7551/mitpress/9780262033589.001.0001>

## Cite this entry

```bibtex
@misc{dictml_transferlearning,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {transfer learning},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/transferlearning.html}
}
```


---

# transformer

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/transformer.html) · [typeset PDF](https://dictionaryofml.org/terms/transformer.pdf) · [LaTeX](https://dictionaryofml.org/terms/transformer.tex) · [Python demo](https://dictionaryofml.org/terms/transformer.py) · [notebook](https://dictionaryofml.org/terms/transformer.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A transformer is an artificial neural network built by composing layers, each of which transforms the features of a data point made up of smaller units, so-called tokens, such as the words of a text or the patches of an image. The feature vectors of the tokens are stacked into a matrix; each layer maps this matrix to one of the same shape, so that layers can be composed freely into a deep artificial neural network. The layers alternate between token mixing, which combines information across tokens, and a position-wise multilayer perceptron, which transforms each token separately.

## Definition

A text consists of words, and an image consists of patches of pixels (see Fig. 1): the data points of many machine learning applications consist of smaller units, so-called tokens. A transformer is an artificial neural network that is built by composing layers, each of which transforms the features of such a token-based data point (Vaswani et al., 2017).

![Figure 1 of the entry transformer](https://dictionaryofml.org/terms/fig/transformer-fig1.svg)

*Figure 1: Two data points that consist of tokens. (a) The opening words of the Universal Declaration of Human Rights (United Nations General Assembly, 1948); each word is one token. (b) A weather-radar-based image of the one-hour precipitation around Krems an der Donau, Austria (August 20, 2026; INCA (Integrated Nowcasting through Comprehensive Analysis) analysis, GeoSphere Austria), with $96 \times 96$ pixels of 1 km; darker pixels indicate more precipitation, and thin gray lines cut the image into $16 \times 16$-pixel patches, each of which is one token. The black line, drawn with a white casing so it stays visible over dark pixels, marks the course of the Danube, and the dot at the image center the location of Krems (river course: OpenStreetMap contributors). Data generated by pythondemos/transformer.py*

The feature vectors (or embeddings) ${\mathbf x}^{(1)}, \ldots, {\mathbf x}^{(n)} \in \mathbb{R}^{d}$ of the $n$ tokens are stacked into a matrix ${\mathbf X} = \big({\mathbf x}^{(1)}, \ldots, {\mathbf x}^{(n)}\big)^{\top}
\in \mathbb{R}^{n \times d}$. Each layer of the original transformer maps such a matrix to a transformed matrix of the same shape, so that layers can be composed freely into a deep artificial neural network (Vaswani et al., 2017, Sect. 3.1). For example, a transformer that translates a sentence transforms the stacked word embeddings, layer by layer, into feature vectors from which the translated words are predicted (Vaswani et al., 2017); transformers also process images by taking patches as tokens (Dosovitskiy et al., 2021). Contemporary LLMs, such as GPT-3, are transformers trained to predict the next token of a text (Brown et al., 2020).

The layers of a transformer alternate between two types of transformations of ${\mathbf X}$ (see Fig. 2). A token-mixing layer maps ${\mathbf X}$ to a new matrix $\widetilde{{\mathbf X}}$, each row of which is a weighted sum of linear images of the rows of ${\mathbf X}$, with weights that can depend on ${\mathbf X}$ itself; the tokens thereby interact. In the original transformer, token mixing is implemented by multi-head attention, whose output for each token is a weighted sum of the value vectors, the rows of ${\mathbf X}$ multiplied by a learned matrix (Vaswani et al., 2017, Sect. 3.2). Attention distinguishes transformers from RNNs, which process the tokens of a data point sequentially instead of in parallel (Vaswani et al., 2017, Sect. 1). A position-wise multilayer perceptron then transforms each row of $\widetilde{{\mathbf X}}$ separately (Vaswani et al., 2017, Sect. 3.3). For each of the two transformations, a residual connection adds the input of the transformation to its output, and a subsequent layer normalization standardizes each row of the resulting matrix (Vaswani et al., 2017, Sect. 3.1).

![Figure 2 of the entry transformer](https://dictionaryofml.org/terms/fig/transformer-fig2.svg)

*Figure 2: One transformer block. Each of the two transformations, multi-head attention for token mixing and a position-wise multilayer perceptron, is bypassed by a residual connection (dashed arrow) whose output is added back and passed through layer normalization. The largest Llama 3 large language model stacks more than 120 of such blocks (Grattafiori and others, 2024, Sect. 3.2)*

The matrices produced by successive layers represent the tokens at increasingly abstract levels. This progression is measured by probing: a simple classifier is trained to predict a chosen property from the output of a single layer, and the depth at which the property becomes predictable locates it in the network. In a language transformer, word-level properties are predictable from the output of early layers. One example is a part-of-speech tag, which marks whether a word is a noun, a verb, or an adjective. Relations between tokens far apart in the text become predictable only in deeper layers (Tenney et al., 2019).

Deep CNNs show the same progression, from edges and textures in early layers to object parts in deeper ones (Zeiler and Fergus, 2014). Vision transformers develop it too: their early layers split attention between nearby and distant patches, while the deeper layers attend across the whole image (Raghu et al., 2021). In language and vision transformers alike, the most informative representations often arise in the middle layers, whose outputs can outperform those of the final layer as features for other tasks (Skean et al., 2025).

Whatever the layers represent, computing them is dominated by the token mixing that produces the matrix of each layer, which is computationally expensive. Attention compares every pair of tokens, so its cost grows as $n^{2}$ with the number $n$ of tokens (Tay et al., 2022, Sect. 2.4). This quadratic growth motivates alternative techniques for token mixing. Some transformer variants reduce the number of tokens in deeper layers (Tay et al., 2022). Any transformation that combines the rows of ${\mathbf X}$ can replace attention: a multilayer perceptron applied across tokens (Tolstikhin et al., 2021), a fixed Fourier transform (Lee-Thorp et al., 2022), or a state-space layer whose cost is linear in $n$ (Gu and Dao, 2024, Sect. 3.3).

**See also:** [attention](https://dictionaryofml.org/terms/attention.html), [artificial neural network](https://dictionaryofml.org/terms/ann.html), layer, token, embedding, word embedding, recurrent neural network, natural language processing, [large language model](https://dictionaryofml.org/terms/llm.html), multilayer perceptron, residual connection, layer normalization, convolutional neural network.

## References

- United Nations General Assembly (1948). Universal Declaration of Human Rights (UDHR). *United Nations*. <https://www.un.org/en/about-us/universal-declaration-of-human-rights>
- Grattafiori and others (2024). The Llama 3 Herd of Models. <https://doi.org/10.48550/arXiv.2407.21783>
- Vaswani et al. (2017). Attention is all you need. *Adv. Neural Inf. Process. Syst.*. <https://papers.nips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html>
- Dosovitskiy et al. (2021). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. *Int. Conf. Learn. Represent. (ICLR)*. <https://openreview.net/forum?id=YicbFdNTTy>
- Brown et al. (2020). Language Models are Few-Shot Learners. *Adv. Neural Inf. Process. Syst.*. <https://proceedings.neurips.cc/paper_files/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html>
- Tenney et al. (2019). BERT Rediscovers the Classical NLP Pipeline. *Proc. 57th Ann. Meeting Assoc. Comput. Linguistics*. <https://doi.org/10.18653/v1/P19-1452>
- Zeiler and Fergus (2014). Visualizing and Understanding Convolutional Networks. *Proc. European Conf. Comput. Vis. (ECCV)*. <https://doi.org/10.1007/978-3-319-10590-1_53>
- Raghu et al. (2021). Do Vision Transformers See Like Convolutional Neural Networks?. *Adv. Neural Inf. Process. Syst.*. <https://proceedings.neurips.cc/paper_files/paper/2021/hash/652cf38361a209088302ba2b8b7f51e0-Abstract.html>
- Skean et al. (2025). Layer by Layer: Uncovering Hidden Representations in Language Models. *Proc. 42nd Int. Conf. Mach. Learn. (ICML)*. <https://openreview.net/forum?id=WGXb7UdvTX>
- Tay et al. (2022). Efficient Transformers: A Survey. *ACM Computing Surveys*. <https://doi.org/10.1145/3530811>
- Tolstikhin et al. (2021). MLP-Mixer: An all-MLP Architecture for Vision. *Adv. Neural Inf. Process. Syst.*. <https://proceedings.neurips.cc/paper_files/paper/2021/hash/cba0a4ee5ccd02fda0fe3f9a3e7b89fe-Abstract.html>
- Lee-Thorp et al. (2022). FNet: Mixing Tokens with Fourier Transforms. *Proc. 2022 Conf. North Amer. Chapter Assoc. Comput. Linguistics: Human Lang. Technologies*. <https://doi.org/10.18653/v1/2022.naacl-main.319>
- Gu and Dao (2024). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. *Conf. Language Modeling*. <https://openreview.net/forum?id=tEYskw1VY2>

## Cite this entry

```bibtex
@misc{dictml_transformer,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {transformer},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/transformer.html}
}
```


---

# transparency

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-10-29. [HTML](https://dictionaryofml.org/terms/transparency.html) · [typeset PDF](https://dictionaryofml.org/terms/transparency.pdf) · [LaTeX](https://dictionaryofml.org/terms/transparency.tex) · [Python demo](https://dictionaryofml.org/terms/transparency.py) · [notebook](https://dictionaryofml.org/terms/transparency.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Transparency is a key requirement for trustworthy artificial intelligence: it names the duties to close the information gaps between the provider of an artificial intelligence system, its deployer, and the persons affected by its predictions. For machine learning methods, transparency is often used interchangeably with explainability: explanations, delivered along with the predictions of a learned hypothesis, let a human user anticipate those predictions. The EU AI Act makes transparency a binding design requirement. Under the Act, the provider must design a high-risk artificial intelligence system so that its deployer can interpret the delivered predictions. Persons interacting with an artificial intelligence system must be informed of that fact. Affected persons may obtain an explanation of decisions based on the predictions delivered by a high-risk artificial intelligence system. The Act also requires documentation of the algorithm design, the training datasets, and the intended use of an artificial intelligence system, as well as a machine-readable marking of synthetic content as artificially generated.

## Definition

Consider a bank that uses an artificial intelligence system to score loan applications. The loan officer sees only the score, and the applicant might not even know that an artificial intelligence system was involved. Transparency names the duties to close such information gaps; it is a fundamental requirement for trustworthy artificial intelligence (High-Level Expert Group on Artificial Intelligence, 2019).

In the context of machine learning methods, transparency is often used interchangeably with explainability (Gallese, 2023): explanations, delivered along with the predictions of a learned hypothesis, let a human user anticipate those predictions (Jung and Nardelli, 2020). Some machine learning methods inherently offer this form of transparency. Classification methods quantify the confidence in an individual prediction via the distance of the feature vector from the decision boundary. A shallow decision tree does not require a separately constructed explanation: presenting the tree as a flow chart of explicit, human-readable decision rules is an explanation of every prediction it delivers (see interpretable machine learning) (Rudin, 2019).

In the broader scope of AI systems, transparency extends beyond explainability: it includes informing persons that an artificial intelligence system is in use and providing information about the limitations, the overall reliability, and the intended use of the system as a whole. The binding transparency obligations discussed below are those of the EU AI Act; other jurisdictions impose related but distinct duties, e.g., the Colorado Automated Decision-Making Technology Act (Colorado General Assembly, 2026) and the court proceedings State v. Loomis on the use of a proprietary recidivism-risk score in sentencing (Supreme Court of Wisconsin, 2016). The Act distributes its obligations among three roles: the provider that develops an artificial intelligence system, the deployer that uses it, and the persons who interact with the artificial intelligence system or are affected by its output (see deployer). The Act's term *output* denotes the predictions delivered by a learned hypothesis; the machine learning term *prediction* is used below. Like interpretability and explainability, which are relative to a specific user or group of users, transparency is relative to its addressee: what must be disclosed depends on the role.

Specifically, toward the deployer, the provider must design and develop a high-risk artificial intelligence system so that its operation is sufficiently transparent to enable the deployers to interpret its predictions and use them appropriately (European Parliament and Council of the European Union, 2024, Art. 13). A second example, medical diagnosis, shows the two roles: the provider is, e.g., a company that develops an X-ray analysis tool, and a clinic that uses the tool acts as the deployer. The provider can meet the duty of Art. 13 by designing the tool to disclose to the clinician the confidence level for the predictions delivered by a learned hypothesis. This duty concerns the meaning of the predictions in the context of their use; the AI Risk Management Framework of the US National Institute of Standards and Technology reserves the term *interpretability* for this contextual meaning (National Institute of Standards and Technology, 2023). This usage differs from interpretability as the comprehension of the computational process of the method; Art. 13 does not require that comprehension.

Toward persons who interact directly with an artificial intelligence system, such as an artificial intelligence-powered chatbot, the provider must design the artificial intelligence system so that these persons are informed of that fact (European Parliament and Council of the European Union, 2024, Art. 50(1)). Providers of AI systems that generate synthetic audio, image, video, or text content must mark such content in a machine-readable format as artificially generated (European Parliament and Council of the European Union, 2024, Art. 50(2)). This marking duty applies whether or not the artificial intelligence system is a high-risk artificial intelligence system; it is waived, e.g., where the artificial intelligence system only performs an assistive function for standard editing (European Parliament and Council of the European Union, 2024, Art. 50(2)). For a deep fake, the deployer must, in addition, disclose visibly that the content is artificially generated or manipulated (European Parliament and Council of the European Union, 2024, Art. 50(4)).

A person can be affected by an artificial intelligence system without directly interacting with it. A patient whose diagnosis is supported by an artificial intelligence-based system deals only with the clinician. Nevertheless, the deployer must inform the patient that a high-risk artificial intelligence system is used concerning them (European Parliament and Council of the European Union, 2024, Art. 26(11)). The EU AI Act also grants the affected person a right to explanation: the deployer must, on request, provide a clear and meaningful explanation of the role of the artificial intelligence system in a decision based on its predictions (European Parliament and Council of the European Union, 2024, Art. 86). In credit scoring, for example, a loan applicant faced with an adverse automated decision may obtain from the deployer, under this right, an explanation of the contributing factors, such as income level or credit history, and use it to contest the decision (see Fig. 1).

Transparency also encompasses documentation detailing the purpose and design choices underlying the artificial intelligence system. For a high-risk artificial intelligence system, the provider must provide documentation that covers the algorithm design and the datasets used for training (European Parliament and Council of the European Union, 2024, Art. 11). The provider of a general-purpose AI model must additionally publish a sufficiently detailed summary of the content used for training (European Parliament and Council of the European Union, 2024, Art. 53). Datasheets for datasets (Gebru et al., 2021) and model cards (Mitchell et al., 2019) help practitioners understand the intended use cases and limitations of an artificial intelligence system.

Concretely, Fig. 1 locates these obligations in a scatterplot of training set data points together with a learned hypothesis, for the credit-scoring example. The solid curve depicts the learned hypothesis $\widehat{h}$ that the artificial intelligence system applies: it maps the income $x$ of an applicant to a predicted credit score $\widehat{h}(x)$, and a loan is approved when the predicted score exceeds an approval threshold (dotted line). The duty of Art. 13 concerns the use of this map: here, the deployer must be able to state the prediction $\widehat{h}(x')$ for an applicant with income $x'$ (open square) and its distance from the threshold.

The filled circles depict the training set $\mathcal{D}^{(\rm train)}$ of completed loans, with income as the feature $x$ and credit score as the label $y$, from which $\widehat{h}$ was learned, e.g., via empirical risk minimization, by minimizing the average loss of the hypothesis over $\mathcal{D}^{(\rm train)}$. The documentation of Art. 11 covers both components of this training: the dataset $\mathcal{D}^{(\rm train)}$ and the design of the algorithm, including the model and the loss used. Income levels outside the range covered by $\mathcal{D}^{(\rm train)}$ (shaded region) are a limitation that the documentation must state, since no estimate of the risk computed from $\mathcal{D}^{(\rm train)}$ covers them.

The right to explanation of Art. 86 is exercised for a single prediction: the applicant with income $x'$, whose predicted score falls below the threshold, may ask why. One answer is a counterfactual: the smallest change of the feature that flips the decision, the income $x''$ (open diamond, arrow) at which $\widehat{h}$ reaches the threshold.

![Figure 1 of the entry transparency](https://dictionaryofml.org/terms/fig/transparency-fig1.svg)

*Figure 1: Transparency obligations of the EU AI Act for a credit-scoring machine learning application. The solid curve depicts the learned hypothesis $\widehat{h}$, whose predictions the deployer must be able to interpret (Art. 13): the prediction $\widehat{h}(x')$ for a loan applicant with income $x'$ (open square) falls below the approval threshold (dotted line). Filled circles are the data points of the training set $\mathcal{D}^{(\rm train)}$ from which $\widehat{h}$ was learned via empirical risk minimization; the provider must document this dataset and the algorithm design (Art. 11), and the shaded range of incomes not covered by $\mathcal{D}^{(\rm train)}$ is a limitation the documentation must state. The applicant is informed that the artificial intelligence system is used (Art. 26(11)) and may obtain an explanation of the decision (Art. 86), e.g., the counterfactual income $x''$ (open diamond, arrow) at which $\widehat{h}$ reaches the threshold*

Fig. 2 summarizes these transparency obligations as information flows between the provider of an artificial intelligence system, its deployer, and the persons exposed to its predictions.

![Figure 2 of the entry transparency](https://dictionaryofml.org/terms/fig/transparency-fig2.svg)

*Figure 2: Transparency obligations of the EU AI Act as information flows, with the roles taken from the medical-diagnosis example. The provider of a high-risk artificial intelligence system supplies technical documentation, covering the algorithm design and the training datasets, along with instructions for use. The deployer must be able to interpret the predictions of the artificial intelligence system. Persons who interact directly with the artificial intelligence system, its users, must be informed of that fact, and generated content must carry a machine-readable mark that it is artificial (Art. 50). An affected person who never interacts with the artificial intelligence system, such as a patient whose diagnosis the system supports, is informed of its use, and owed an explanation of an adverse decision, by the deployer (Arts. 26(11), 86)*

**See also:** [trustworthy artificial intelligence](https://dictionaryofml.org/terms/trustAI.html), [explainability](https://dictionaryofml.org/terms/explainability.html), [interpretability](https://dictionaryofml.org/terms/interpretability.html), [interpretable machine learning](https://dictionaryofml.org/terms/interpretableml.html), [explanation](https://dictionaryofml.org/terms/explanation.html), counterfactual, right to explanation, EU AI Act, high-risk artificial intelligence system, artificial intelligence system, general-purpose AI model, provider, deployer, watermarking, deep fake, content provenance.

## References

- High-Level Expert Group on Artificial Intelligence (2019). Ethics Guidelines for Trustworthy AI. <https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai>
- Gallese (2023). The AI Act proposal: A new right to technical interpretability?. *SSRN Electron. J.*. <https://doi.org/10.2139/ssrn.4398206>
- Jung and Nardelli (2020). An Information-Theoretic Approach to Personalized Explainable Machine Learning. *IEEE Signal Process. Lett.*. <https://doi.org/10.1109/lsp.2020.2993176>
- Rudin (2019). Stop explaining black box machine learning models for high-stakes decisions and use interpretable models instead. *Nature Mach. Intell.*. <https://doi.org/10.1038/s42256-019-0048-x>
- Colorado General Assembly (2026). Senate Bill 26-189 --- Automated Decision-Making Technology. <https://leg.colorado.gov/bills/sb26-189>
- Supreme Court of Wisconsin (2016). State v. Loomis, 2016 WI 68, 881 N.W.2d 749. <https://www.wicourts.gov/sc/opinion/DisplayDocument.pdf?content=pdf&seqNo=171690>
- European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act) (Text with EEA relevance). <https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng>
- National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). <https://doi.org/10.6028/NIST.AI.100-1>
- Gebru et al. (2021). Datasheets for datasets. *Commun. ACM*. <https://doi.org/10.1145/3458723>
- Mitchell et al. (2019). Model Cards for Model Reporting. *Proc. Conf. Fairness, Accountability, Transparency*. <https://doi.org/10.1145/3287560.3287596>

## Cite this entry

```bibtex
@misc{dictml_transparency,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {transparency},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/transparency.html}
}
```


---

# trustworthy artificial intelligence (trustworthy AI)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-07. [HTML](https://dictionaryofml.org/terms/trustAI.html) · [typeset PDF](https://dictionaryofml.org/terms/trustAI.pdf) · [LaTeX](https://dictionaryofml.org/terms/trustAI.tex) · [Python demo](https://dictionaryofml.org/terms/trustAI.py) · [notebook](https://dictionaryofml.org/terms/trustAI.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Trustworthy artificial intelligence (trustworthy AI) names an artificial intelligence system that meets requirements beyond delivering accurate predictions. From an engineering perspective, the ultimate goal of machine learning is to predict the label of any data point accurately, i.e., to learn a hypothesis with small risk. This perspective is too narrow, since a machine learning method is embedded in a sociotechnical system that includes human users. Trustworthy AI therefore formulates additional requirements for the method: human oversight, robustness, privacy, transparency, fairness, societal well-being, and accountability. Guidelines and regulations across the world state these requirements, from the OECD AI Principles to national rules in China and Australia and federal and state rules in the US, and the Ethics Guidelines for Trustworthy AI of the European Commission are one example among them. Their seven key requirements are stated for the artificial intelligence system as a whole, including the people and the organization around the machine learning method inside it. The requirements translate into design choices for the components of empirical risk minimization: the training set, the model, the loss function, and the algorithm. For example, robustness requires a model that contains no hypothesis map that is too sensitive to a small perturbation of the feature vector. Each such requirement is a measurable property of the learned hypothesis or of the method that learned it: a bound on the change of a prediction under a perturbation of the feature vector, the change of the prediction when only a sensitive attribute is flipped, the counterfactual that explains a rejection, or the share of borderline decisions passed to a person.

## Definition

Consider a bank that uses an artificial intelligence system to decide on loan applications. The system applies a hypothesis $\widehat{h}$, learned from the records of completed loans, to the feature vector ${\mathbf x}$ of an applicant, which collects the income, the gender, and further attributes. The loan is approved when the predicted creditworthiness $\widehat{h}({\mathbf x})$ exceeds a threshold $\tau$.

From an engineering perspective, the ultimate goal of a machine learning method is to predict the label of any data point accurately. This goal is formalized as a small risk of $\widehat{h}$, the expected loss on an applicant drawn from the same probability distribution as the completed loans. This perspective is too narrow. The machine learning method is embedded in a sociotechnical system that includes human users: the applicants whose loans it decides, the loan officers who act on its predictions, and the supervisor who audits the bank. These users rely on the decisions only if the system meets requirements beyond the accuracy of its predictions. A small change of the income must not flip the decision, the gender alone must not flip it either, a rejected applicant must learn why, and a loan officer must be able to overrule the system. Trustworthy AI formulates such requirements for a machine learning method and names an artificial intelligence system that meets them.

Such requirements are stated by guidelines and regulations adopted across the world since 2019. Examples are the OECD AI Principles (OECD, 2019), Australia's AI Ethics Principles (Department of Industry, Science, Energy and Resources, 2024), the AI Risk Management Framework of the US National Institute of Standards and Technology (National Institute of Standards and Technology, 2023), China's Interim Measures for generative artificial intelligence services (Cyberspace Administration of China, 2023), and the management-system standard ISO/IEC 42001 (International Organization for Standardization and International Electrotechnical Commission, 2023). These documents name much the same requirements and differ in their legal force. The Ethics Guidelines for Trustworthy AI of the European Commission's High-Level Expert Group (High-Level Expert Group on Artificial Intelligence, 2019) are used as the example here, since they enumerate the requirements explicitly. Their seven key requirements (KRs) are human agency and oversight (KR1), technical robustness and safety (KR2), privacy and data governance (KR3), transparency (KR4), diversity, nondiscrimination, and fairness (KR5), societal and environmental well-being (KR6), and accountability (KR7). The accompanying assessment list turns each requirement into questions that the provider and the deployer of the artificial intelligence system answer about their system (High-Level Expert Group on Artificial Intelligence, 2020).

These requirements are stated for an artificial intelligence system as a whole, which includes the data pipeline, the user interface, and the organization that operates it, and not only for the machine learning method inside it. KR6 and KR7 concern the purpose and the governance of the system and have no counterpart in the machine learning method. KR1 to KR5, in contrast, translate into design choices for the components of empirical risk minimization: the training set $\mathcal{D}^{(\rm train)}$, the model $\mathcal{H}$, the loss function, and an algorithm that minimizes the average loss over $\mathcal{D}^{(\rm train)}$ (Pfau and Jung, 2024). KR2, e.g., requires a model $\mathcal{H}$ that contains no hypothesis map that is too sensitive to a small perturbation of the feature vector. Excluding such maps from $\mathcal{H}$ is model pruning, one of the three routes that regularization distinguishes. Each of these requirements is a measurable property of the learned hypothesis $\widehat{h}$ or of the method that learned it.

Fig. 1 places KR1, KR2, KR4, and KR5 in the credit-scoring example. The horizontal axis is a one-dimensional sketch of the feature vector ${\mathbf x}$: the income, the gender, and the further attributes of an applicant collapsed into one coordinate. The vertical axis is the creditworthiness, the label $y$. The filled circles are the training set $\mathcal{D}^{(\rm train)}$ of $m = 5$ completed loans, the solid curve a hypothesis $\widehat{h}$ drawn for illustration in place of one learned from $\mathcal{D}^{(\rm train)}$, and the dotted line the approval threshold $\tau = 3.8$. The applicant with feature vector ${\mathbf x}'$ (open square) obtains the predicted creditworthiness $\widehat{h}({\mathbf x}') =
2.94$ and is rejected.

KR2 asks that a small perturbation of the feature vector changes the prediction little (Pfau and Jung, 2024, Sect. V). The perturbation of ${\mathbf x}'$ by $\delta = 0.5$ (horizontal arrow) moves the predicted creditworthiness from $2.94$ to $3.19$ (vertical arrow). The slope of $\widehat{h}$ never exceeds $1.225$, so the hypothesis is Lipschitz continuous with that constant: no perturbation by $\delta$ changes the prediction by more than $1.225\,\delta$ (see robustness). KR2 also covers perturbations of $\mathcal{D}^{(\rm train)}$, e.g., by data poisoning, against which the stability of the machine learning method protects.

KR5 asks that the decision does not depend on a sensitive attribute $s$ such as gender. The open triangle depicts the same applicant with only the gender flipped, the feature vector ${\mathbf x}^{\star}$. The predicted creditworthiness rises to $\widehat{h}({\mathbf x}^{\star}) = 3.93$, above $\tau$, so the system approves the loan (as a borderline case, it is also passed to the loan officer): the decision depends on the gender, which violates the counterfactual criterion of fairness. The gap $\widehat{h}({\mathbf x}^{\star}) -
\widehat{h}({\mathbf x}') = 0.99$ measures the violation. It is closed by a design choice for a component of empirical risk minimization: the model (excluding the sensitive attribute and the features correlated with it), the loss function (a penalty term for the gap), or $\mathcal{D}^{(\rm train)}$ (resampling the groups) (Pfau and Jung, 2024, Sect. VIII).

KR4 asks for transparency: the rejected applicant is told that an artificial intelligence system was used and may obtain an explanation. The counterfactual ${\mathbf x}''$ (open diamond), at which $\widehat{h}$ reaches $\tau$, is such an explanation. It is the smallest change of the income, at fixed gender, that flips the decision, and it tells the applicant the income at which the loan would be approved. The gender flip ${\mathbf x}^{\star}$ also flips the decision, but it is not a change the applicant can make, which is what makes it a violation of KR5 rather than an explanation.

KR1 asks that a person can oversee and overrule the decisions. The shaded band marks predicted creditworthiness within $\varepsilon = 0.3$ of $\tau$. Applications in this band are passed to a loan officer, who decides; one of the five loans in $\mathcal{D}^{(\rm train)}$ falls into it. A wider band passes more applications to the officer and leaves fewer borderline decisions to the system (see human oversight).

Finally, KR3 has no place in the figure, since it concerns the data rather than a prediction: it asks that the records of completed loans, which are personal data, are protected during training and cannot be recovered from $\widehat{h}$; differential privacy and federated learning are design choices for the algorithm and for $\mathcal{D}^{(\rm train)}$ to that end (see data governance).

![Figure 1 of the entry trustAI](https://dictionaryofml.org/terms/fig/trustAI-fig1.svg)

*Figure 1: Four key requirements for trustworthy AI in the credit-scoring example. The horizontal axis is a one-dimensional sketch of the feature vector. The solid curve is a hypothesis $\widehat{h}$ drawn for illustration beside the training set $\mathcal{D}^{(\rm train)}$ of five completed loans (filled circles); a loan is approved when the predicted creditworthiness exceeds the threshold $\tau$ (dotted line). The applicant with feature vector ${\mathbf x}'$ (open square) is rejected. KR1: predictions in the shaded band are passed to a loan officer. KR2: a perturbation of ${\mathbf x}'$ by $\delta$ (horizontal arrow) changes the prediction little (vertical arrow). KR4: the counterfactual ${\mathbf x}''$ (open diamond), the smallest change of the feature vector at which $\widehat{h}$ reaches $\tau$, explains the rejection. KR5: the same applicant with the gender flipped, ${\mathbf x}^{\star}$ (open triangle), is scored above $\tau$, so the decision depends on the gender; as the score lies in the band, this case too is passed to the loan officer*

Beyond guidelines, binding law states the same requirements. The EU AI Act is one example: it makes KR1 to KR4 obligatory for a high-risk artificial intelligence system, a class that includes the credit scoring of natural persons (European Parliament and Council of the European Union, 2024, Annex III, Arts. 9, 10, and 13–15). China's Interim Measures bind the providers of generative artificial intelligence services to transparency and nondiscrimination (Cyberspace Administration of China, 2023). In the US, individual states legislate. California gives a consumer subject to an automated lending decision a notice, an opt-out, and access to the logic of the decision (California Privacy Protection Agency, 2025). Arizona requires a licensed health care provider to review every denial of an insurance claim that artificial intelligence would issue on grounds of medical judgment (Arizona Legislature, 2025).

These requirements can conflict: closing the gender gap raises the average loss on $\mathcal{D}^{(\rm train)}$, and the noise that differential privacy adds to the training makes explanations less precise (Pfau and Jung, 2024). Trustworthiness is therefore a set of measured properties that are balanced for the application, not a single number.

**See also:** artificial intelligence system, EU AI Act, high-risk artificial intelligence system, provider, deployer, human oversight, robustness, stability, data poisoning, data governance, differential privacy, [transparency](https://dictionaryofml.org/terms/transparency.html), [explainability](https://dictionaryofml.org/terms/explainability.html), counterfactual, [fairness](https://dictionaryofml.org/terms/fairness.html), sensitive attribute, [empirical risk minimization](https://dictionaryofml.org/terms/erm.html).

## References

- OECD (2019). Recommendation of the Council on Artificial Intelligence. <https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0449>
- Department of Industry, Science, Energy and Resources (2024). Australia's AI Ethics Principles. <https://www.industry.gov.au/publications/australias-artificial-intelligence-ethics-framework/australias-ai-ethics-principles>
- National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). <https://doi.org/10.6028/NIST.AI.100-1>
- Cyberspace Administration of China (2023). Interim Measures for the Management of Generative Artificial Intelligence Services. <https://www.chinalawtranslate.com/en/generative-ai/>
- International Organization for Standardization and International Electrotechnical Commission (2023). ISO/IEC 42001:2023 --- Information technology --- Artificial intelligence --- Management system. <https://www.iso.org/standard/42001>
- High-Level Expert Group on Artificial Intelligence (2019). Ethics Guidelines for Trustworthy AI. <https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai>
- High-Level Expert Group on Artificial Intelligence (2020). The Assessment List for Trustworthy Artificial Intelligence (ALTAI): For self assessment. *Publications Office*. <https://doi.org/10.2759/002360>
- Pfau and Jung (2024). Engineering Trustworthy AI: A Developer Guide for Empirical Risk Minimization. *arXiv preprint arXiv:2410.19361*. <https://arxiv.org/abs/2410.19361>
- European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act) (Text with EEA relevance). <https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng>
- California Privacy Protection Agency (2025). CCPA Updates, Insurance, Cybersecurity Audits, Risk Assessments, and Automated Decisionmaking Technology (ADMT) Regulations. <https://cppa.ca.gov/regulations/>
- Arizona Legislature (2025). Claims; Prior Authorization; Conduct. <https://www.azleg.gov/legtext/57leg/1r/bills/hb2175h.htm>

## Cite this entry

```bibtex
@misc{dictml_trustAI,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {trustworthy artificial intelligence (trustworthy AI)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/trustAI.html}
}
```


---

# unsupervised learning

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-04. [HTML](https://dictionaryofml.org/terms/unsupervisedlearning.html) · [typeset PDF](https://dictionaryofml.org/terms/unsupervisedlearning.pdf) · [LaTeX](https://dictionaryofml.org/terms/unsupervisedlearning.tex) · [Python demo](https://dictionaryofml.org/terms/unsupervisedlearning.py) · [notebook](https://dictionaryofml.org/terms/unsupervisedlearning.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Unsupervised learning refers to machine learning methods that do not require knowledge of the label of any data point. These methods learn a hypothesis by loss functions that are constructed from the features of data points only. Two important examples of unsupervised learning are clustering and dimensionality reduction. Clustering methods use some notion of clustering error to learn model parameters that encode a cluster structure. Dimensionality reduction methods construct a loss from the reconstruction error.

## Definition

Twenty days at Krems an der Donau are each characterized by two features, the daily minimum and the daily maximum temperature, and nothing else is recorded about them (Fig. 1). Unsupervised learning methods learn a hypothesis map that predicts a label that is not known for any of these data points. For example, the label could be an index that selects one out of two clusters to which the data point belongs. Another label could be the optimal scalar representation of a data point.

Formally, an unsupervised machine learning method learns a hypothesis out of a hypothesis space based on an unlabeled dataset $\mathcal{D} = \big\{ {\mathbf x}^{(1)}, \ldots,
{\mathbf x}^{(m)} \big\}$ of feature vectors ${\mathbf x}^{(r)} \in \mathcal{X}$, with no label space. The model parameters are typically learned by an instance of empirical risk minimization, minimizing the average loss over the dataset,

$$
\widehat{h} \in \mathop{\mathrm{arg\,min}}_{h \in \mathcal{H}}
\frac{1}{m} \sum_{r=1}^{m}
L\left({\mathbf x}^{(r)},h \right)
\text{.}
$$

What sets unsupervised methods apart from supervised methods, such as linear regression, is the choice of loss function.

Two important families of unsupervised methods are obtained by specific constructions of the loss, clustering and dimensionality reduction. The clustering method k-means uses $k$ cluster centroids as model parameters ${\mathbf w} = \big\{ {{\mathbf w}}^{(1)}, \ldots,
{{\mathbf w}}^{(k)} \big\}$ (Hastie et al., 2009, Sect. 14.3.6). They are learned by minimizing the average clustering error, the loss

$$
L\left({\mathbf x},{\mathbf w} \right)
= \min_{c = 1, \ldots, k}
\left\lVert {\mathbf x} - {{\mathbf w}}^{(c)} \right\rVert_{2}^{2}
\text{.}
$$

Principal component analysis is a dimensionality reduction method that learns a linear feature transformation ${\mathbf x} \mapsto {\mathbf W} {\mathbf x}$ together with a reconstruction map ${\mathbf z} \mapsto {\mathbf R} {\mathbf z}$ that rebuilds the feature vector from it. Both are model parameters, and for a given ${\mathbf R}$ the loss of a data point is its reconstruction error

$$
L\left({\mathbf x},\left( {\mathbf W},{\mathbf R} \right) \right)
= \left\lVert {\mathbf x} - {\mathbf R} {\mathbf W} {\mathbf x} \right\rVert_{2}^{2}
\text{.}
$$

One ${\mathbf R}$ is used for the whole dataset, and empirical risk minimization learns it together with ${\mathbf W}$ by minimizing the average of this loss (Hastie et al., 2009, Sect. 14.5.1). For the learned pair the reconstruction is the orthogonal projection of the feature vector on a subspace of dimension $d'$. Fig. 1 shows both on the Krems days: the two cluster centroids with the data points assigned to each, and the learned direction with the orthogonal projections that give each day its single number.

![Figure 1 of the entry unsupervisedlearning](https://dictionaryofml.org/terms/fig/unsupervisedlearning-fig1.svg)

*Figure 1: Twenty weather recordings at Krems an der Donau, each a data point with two features and no label. The clustering method k-means learned the two cluster centroids (filled squares), and the solid ellipses enclose the data points assigned to each. The dashed line carries the direction learned by a linear feature transformation. The filled triangles mark the orthogonal projections, which represent each data point by a single number, the location of the triangle along the dashed line. No label entered either computation. Data generated by pythondemos/unsupervisedlearning.py*

**See also:** [clustering](https://dictionaryofml.org/terms/clustering.html), [$k$-means](https://dictionaryofml.org/terms/kmeans.html), cluster centroid, [dimensionality reduction](https://dictionaryofml.org/terms/dimred.html), [principal component analysis](https://dictionaryofml.org/terms/pca.html), feature transformation, [feature learning](https://dictionaryofml.org/terms/featlearn.html), [label](https://dictionaryofml.org/terms/label.html), [dataset](https://dictionaryofml.org/terms/dataset.html), [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), [loss](https://dictionaryofml.org/terms/loss.html), [supervised learning](https://dictionaryofml.org/terms/supervisedlearning.html), [self-supervised learning](https://dictionaryofml.org/terms/selfsupervisedlearning.html), semi-supervised learning, clustering error.

## References

- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>

## Cite this entry

```bibtex
@misc{dictml_unsupervisedlearning,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {unsupervised learning},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/unsupervisedlearning.html}
}
```


---

# validation error

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-11. [HTML](https://dictionaryofml.org/terms/valerr.html) · [typeset PDF](https://dictionaryofml.org/terms/valerr.pdf) · [LaTeX](https://dictionaryofml.org/terms/valerr.tex) · [Python demo](https://dictionaryofml.org/terms/valerr.py) · [notebook](https://dictionaryofml.org/terms/valerr.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

The validation error $E_{v}$ of a learned hypothesis is its average loss over a validation set. For a fixed hypothesis and data points that are realizations of independent and identically distributed RVs, the validation error is an unbiased estimator of the risk. If the training set is itself modelled as random, the learned hypothesis becomes a random variable too, and the validation error is unbiased conditioned on the learned hypothesis. The validation error characterizes a learned hypothesis that some machine learning method obtained from a training set. It can therefore be used to choose between different models or, more generally, between different machine learning methods. The validation error also helps to diagnose a machine learning method and to guide its revision.

## Definition

The weather station Krems an der Donau (Austria) recorded the morning minimum and the maximum temperature of every day from 2020 to 2024. Each day is a data point: its feature $x$ is the morning minimum temperature, its label $y$ the maximum temperature of the day. Fig. 1 shows twenty of them as a scatterplot and singles out six. Three of the days, drawn in black, form the training set. The hypothesis $\widehat{h}^{(2)}$ learned from them is a degree-two polynomial through all three days, so its training error is zero. Three further days, drawn as red triangles, form a validation set. The dotted bars are the prediction errors of the polynomial on them, the differences between the recorded maximum and the value of the curve. The average of the three squared bars is the validation error of the polynomial, $42.58$.

![Figure 1 of the entry valerr](https://dictionaryofml.org/terms/fig/valerr-fig1.svg)

*Figure 1: Twenty of the $1827$ days at Krems an der Donau, 2020 to 2024 (every $92$nd day), each a gray dot with the morning minimum temperature as its feature $x$ and the maximum temperature of the day as its label $y$. The three black dots form the training set, and the dashed curve is the degree-two polynomial $\widehat{h}^{(2)}$ learned from them; it passes through all three, so its training error is zero. The three red triangles indicate days that form the validation set, and the dotted bars are the prediction errors of the polynomial on them. The average of the three squared bars is the validation error. Data generated by pythondemos/valerr.py*

The validation error is the average loss of $\widehat{h}^{(2)}$ on days that did not belong to the training set. The training error carries no information about such days: the polynomial passes through the three training days by construction. The validation error is close to the average squared error loss of $\widehat{h}^{(2)}$ over all days of the five years, $42.90$. A hypothesis with a larger training error can be the better one: the linear hypothesis that empirical risk minimization delivers from the same three days over the linear model has a training error of $7.91$ and a validation error of $6.98$.

The validation error $E_{v}$ of a hypothesis $\widehat{h}$ is its average loss over a validation set $\mathcal{D}^{(\rm val)}$,

$$
E_{v} = \frac{1}{|\mathcal{D}^{(\rm val)}|} \sum_{{\mathbf z} \in \mathcal{D}^{(\rm val)}}
L\left({\mathbf z},\widehat{h} \right) \text{.}
$$

Because no data point of $\mathcal{D}^{(\rm val)}$ entered the training, $E_{v}$ estimates the risk of $\widehat{h}$. The training error, computed over the training set whose average loss empirical risk minimization minimized, is below it on average: the same data points both selected $\widehat{h}$ and assessed it (Hastie et al., 2009, Sect. 7.4).

The validation error is an average of $|\mathcal{D}^{(\rm val)}|$ losses. For data points that are realizations of independent and identically distributed RVs, $E_{v}$ is the realization of a random variable whose expectation is the risk of $\widehat{h}$, since each summand has that expectation. The validation error is therefore an unbiased estimator of that risk. Its variance is the variance of a single loss divided by $|\mathcal{D}^{(\rm val)}|$, so its standard deviation decreases as $|\mathcal{D}^{(\rm val)}|$ grows.

In practice $\widehat{h} = \mathcal{A}(\mathcal{D}^{(\rm train)})$ is the output of a training method, a map $\mathcal{A}$ that reads in the training set $\mathcal{D}^{(\rm train)}$, so it is itself the realization of a random variable: a different realization of $\mathcal{D}^{(\rm train)}$ would have delivered a different hypothesis. The validation error is then unbiased conditioned on the learned hypothesis. If the data points of $\mathcal{D}^{(\rm val)}$ are independent of those of $\mathcal{D}^{(\rm train)}$, and hence of $\widehat{h}$ (Shalev-Shwartz and Ben-David, 2014, Sect. 11.2.1), the conditional expectation of $E_{v}$ given $\widehat{h}$ is the risk of that hypothesis,

$$
\mathbb{E} \big\{ E_{v} \mid \widehat{h} \big\}
= \bar{L} \big( \widehat{h} \big) \text{.}
$$

The training error has no such property: conditioned on $\widehat{h}$, the data points of $\mathcal{D}^{(\rm train)}$ are no longer independent and identically distributed realizations of the probability distribution of a data point, because $\widehat{h}$ was chosen to make its average loss on them small.

A guarantee for the validation error concerns one hypothesis only: for a loss with values in $[0,1]$, Hoeffding's inequality gives $|E_{v} - \bar{L} \big( \widehat{h} \big)| \leq
\sqrt{\ln(2/\delta) / (2 |\mathcal{D}^{(\rm val)}|)}$ with probability at least $1-\delta$, with no term for the hypothesis space $\mathcal{H}$ (Shalev-Shwartz and Ben-David, 2014, Thm. 11.1). A guarantee for the training error of the empirical risk minimization solution must hold uniformly over $\mathcal{H}$, since empirical risk minimization compares all its hypotheses (Shalev-Shwartz and Ben-David, 2014, Sects. 4.1 and 11.2.1).

A numerical experiment shows how much the validation error varies when the validation set is chosen at random. Consider the hypothesis

$$
\widehat{h}(x) = 10.47 + 1.05 \, x \text{,}
$$

obtained by linear regression from forty data points of the same station record, every ninth day of 2024. The data points of $306$ further days of that year, none of them used for training, are collected in a dataset, the test set $\mathcal{D}^{(\rm test)}$. Here $\mathcal{D}^{(\rm test)}$ serves only as the pool from which each validation set is drawn at random; its own role, assessing a selected hypothesis, is taken up at the end.

Fig. 2 shows the value range of the validation error of $\widehat{h}$ over repeated draws of a validation set of prescribed size from $\mathcal{D}^{(\rm test)}$. The standard deviation of the validation error decreases with increasing size of the validation set. The decrease follows the $1/\sqrt{|\mathcal{D}^{(\rm val)}|}$ form of the bound from Hoeffding's inequality above, though the bound itself does not apply, since the squared error loss is not confined to $[0,1]$.

![Figure 2 of the entry valerr](https://dictionaryofml.org/terms/fig/valerr-fig2.svg)

*Figure 2: Spread of the validation error computed for the hypothesis $\widehat{h}(x) = 10.47 + 1.05 \, x$. The validation errors are computed for different validation sets that are, for a given size, randomly selected from $\mathcal{D}^{(\rm test)}$. The dataset $\mathcal{D}^{(\rm test)}$ consists of data points that have not been used to learn $\widehat{h}$. Each bar spans one standard deviation either side of the mean. The dashed curve is the average loss computed over all of $\mathcal{D}^{(\rm test)}$. The standard deviation decreases as $|\mathcal{D}^{(\rm val)}|$ grows. Data generated by pythondemos/valerr.py*

Beyond estimating the risk of one hypothesis, the validation error is compared with the training error and across models. A training error far below the validation error indicates overfitting (Hastie et al., 2009, Sect. 7.2). Comparing the validation errors of several candidate models is model selection (Bishop, 2006, Sect. 1.3). A model has no validation error of its own. The phrase is short for the validation error of the hypothesis that $\mathcal{A}$ delivers from that model, since $E_{v}$ is the average loss of a single hypothesis. The selected model is the one whose hypothesis attains the smallest of the compared validation errors, so that validation error is biased downward as an estimate of the risk, and a test set is kept apart for the final assessment (Bishop, 2006, Sect. 1.3; Hastie et al., 2009, Sect. 7.2).

**See also:** [validation set](https://dictionaryofml.org/terms/valset.html), validation, [training set](https://dictionaryofml.org/terms/trainset.html), [test set](https://dictionaryofml.org/terms/testset.html), [training error](https://dictionaryofml.org/terms/trainerr.html), [loss](https://dictionaryofml.org/terms/loss.html), risk, [hypothesis](https://dictionaryofml.org/terms/hypothesis.html), [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), [overfitting](https://dictionaryofml.org/terms/overfitting.html), [model selection](https://dictionaryofml.org/terms/modelsel.html), generalization gap, conditional expectation, [independent and identically distributed assumption](https://dictionaryofml.org/terms/iidasspt.html), unbiased estimator.

## References

- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>

## Cite this entry

```bibtex
@misc{dictml_valerr,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {validation error},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/valerr.html}
}
```


---

# validation set

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-03. [HTML](https://dictionaryofml.org/terms/valset.html) · [typeset PDF](https://dictionaryofml.org/terms/valset.pdf) · [LaTeX](https://dictionaryofml.org/terms/valset.tex) · [Python demo](https://dictionaryofml.org/terms/valset.py) · [notebook](https://dictionaryofml.org/terms/valset.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A validation set consists of data points that have not been used for model training. For a learned hypothesis, the average loss on the validation set, the validation error, indicates how well the hypothesis predicts the labels of data points outside the training set. The validation error is used for model selection: the hypotheses learned by different machine learning methods are compared, and the one with the smallest validation error is selected. The selected hypothesis depends on the validation set, so assessing it requires a test set whose data points entered neither training nor selection. The size of the validation set determines how reliable the validation error is. When data points are scarce, k-fold cross-validation uses each fold of the dataset in turn as validation set and averages the $k$ validation errors.

## Definition

Consider three days of weather recordings: for each day, the morning minimum temperature is the feature and the maximum daytime temperature is the label. Fig. 1 shows a training set $\mathcal{D}^{(\rm train)}$ of three such days along with two hypotheses learned from it: the linear hypothesis $\widehat{h}^{(1)}$ delivered by empirical risk minimization on the linear model, and a degree-two polynomial $\widehat{h}^{(2)}$ that passes through all three data points and therefore attains a training error of zero.

Three further days, marked by open triangles, were held back from model training: they form a validation set $\mathcal{D}^{(\rm val)}$. The validation error of a hypothesis $\widehat{h}$ learned from the training set is its average loss on the validation set (Shalev-Shwartz and Ben-David, 2014, Sect. 11.2.1),

$$
\frac{1}{|\mathcal{D}^{(\rm val)}|} \sum_{{\mathbf z} \in \mathcal{D}^{(\rm val)}}
L\left({\mathbf z},\widehat{h} \right) \text{.}
$$

Here, $L\left({\mathbf z},\widehat{h} \right)$ is the loss of $\widehat{h}$ on the data point ${\mathbf z}$. In Fig. 1, the polynomial $\widehat{h}^{(2)}$ attains a training error of zero but a validation error of $17.3$, while the linear hypothesis $\widehat{h}^{(1)}$ has a training error of $2.9$ and a validation error of $3.0$.

![Figure 1 of the entry valset](https://dictionaryofml.org/terms/fig/valset-fig1.svg)

*Figure 1: Data points representing daily weather conditions. Each data point is a day, with the morning minimum temperature as its feature $x$ and the maximum daytime temperature as its label $y$. The three filled circles form the training set $\mathcal{D}^{(\rm train)}$; the solid curve is the linear hypothesis $\widehat{h}^{(1)}$ and the dashed polynomial the hypothesis $\widehat{h}^{(2)}$, both learned from $\mathcal{D}^{(\rm train)}$. The three open triangles were held back from model training and form a validation set $\mathcal{D}^{(\rm val)}$: the average loss on them, $3.0$ for $\widehat{h}^{(1)}$ and $17.3$ for $\widehat{h}^{(2)}$, is the validation error of each hypothesis. Data generated by pythondemos/valset.py*

The comparison of the two validation errors is an instance of model selection: the hypotheses learned by different machine learning methods are compared, and the one with the smallest validation error is selected (Bishop, 2006, Sect. 1.3). In Fig. 1, the hypothesis $\widehat{h}^{(1)}$ is selected. The selected hypothesis depends on $\mathcal{D}^{(\rm val)}$: its validation error is the smallest of the compared validation errors, so the validation error of the selected hypothesis is biased downward.

Assessing the hypothesis learned from the model that was chosen on the basis of the validation errors requires a third dataset whose data points entered neither training nor selection: the test set $\mathcal{D}^{(\rm test)}$ (Hastie et al., 2009, Sect. 7.2). In Fig. 2, the three days of the test set, marked by open squares, yield a test error $E^{(\rm test)} = 1.3$ for the selected hypothesis $\widehat{h}^{(1)}$. The test error is the only one of the three numbers that was not used to choose anything, so it is not biased by selection. With only three test days, however, it is itself a noisy estimate of the risk of $\widehat{h}^{(1)}$.

![Figure 2 of the entry valset](https://dictionaryofml.org/terms/fig/valset-fig2.svg)

*Figure 2: Model selection with a validation set. The thick solid curve is the selected hypothesis $\widehat{h}^{(1)}$ and the dashed polynomial the rejected hypothesis $\widehat{h}^{(2)}$, both from Fig. 1; the validation error over the validation set $\mathcal{D}^{(\rm val)}$ (open triangles) selects the hypothesis $\widehat{h}^{(1)}$ ($3.0$ against $17.3$ for the polynomial $\widehat{h}^{(2)}$). The test set $\mathcal{D}^{(\rm test)}$ (open squares) assesses the selected hypothesis, which incurs a test error $E^{(\rm test)} = 1.3$ on it. Data generated by pythondemos/valset.py*

The validation error is an average of $|\mathcal{D}^{(\rm val)}|$ losses, one per validation data point. How closely it tracks the risk of a learned hypothesis therefore depends on the size $|\mathcal{D}^{(\rm val)}|$, and a validation set that is too small gives an unreliable validation error (Bishop, 2006, Sect. 1.3).

Take a loss with values in $[0,1]$, a fixed hypothesis $\widehat{h}$, and a confidence parameter $\delta$ in $(0, 1)$ chosen by the user. Hoeffding's inequality (Shalev-Shwartz and Ben-David, 2014, Thm. 11.1) then guarantees, with probability at least $1 - \delta$, that the risk of $\widehat{h}$ lies within an uncertainty band of half-width $\epsilon$ around the measured validation error. The guarantee requires that

$$
|\mathcal{D}^{(\rm val)}| \geq \frac{\ln(2/\delta)}{2 \epsilon^{2}} \text{.}
$$

Fig. 3 shows this lower bound as a function of the half-width $\epsilon$ for three values of $\delta$; it holds for any validation error, not only those of the weather example. Narrowing the band is costly: halving $\epsilon$ quadruples the required size. To have the validation error within $\epsilon = 0.1$ of the risk with $\delta = 0.1$, it suffices to hold back $|\mathcal{D}^{(\rm val)}| \geq 150$ data points, the point of the dashed curve at $\epsilon = 0.1$ in Fig. 3, regardless of the hypothesis space and of the size of $\mathcal{D}^{(\rm train)}$. The bound covers one fixed hypothesis. A bound that covers the selected hypothesis must hold for all compared hypotheses at once, which requires a larger validation set (Shalev-Shwartz and Ben-David, 2014, Thm. 11.2).

![Figure 3 of the entry valset](https://dictionaryofml.org/terms/fig/valset-fig3.svg)

*Figure 3: The lower bound $\ln(2/\delta)/(2 \epsilon^{2})$ on the size $|\mathcal{D}^{(\rm val)}|$ as a function of the half-width $\epsilon$ of the uncertainty band around the measured validation error, for three values of the confidence parameter $\delta$: $0.01$ (solid curve), $0.1$ (dashed curve), and $0.5$ (dotted curve). The vertical axis is logarithmic. Halving $\epsilon$ from $0.1$ to $0.05$ quadruples the required size (filled circles on the dashed curve, $150$ to $600$), while lowering the confidence parameter $\delta$ from $0.5$ to $0.01$ at $\epsilon = 0.1$ raises it by a factor around four (open circles, $70$ to $265$)*

The bound can be checked by simulation. Fig. 4 reports how often the validation error of a fixed hypothesis deviates from the risk by more than $\epsilon$, when the validation set holds exactly the number of data points that the bound requires. The loss used there takes the values $0$ and $1$ with equal probability, so its variance is the largest a loss with values in $[0,1]$ can have. Each share is measured over $200{,}000$ validation sets. Every measured share stays below its $\delta$, and below $\delta / 4$, so the size the bound requires is larger than the size the measured deviations call for. A validation set eight times smaller does fail the guarantee. At $\epsilon = 0.1$ and $\delta = 0.1$ the bound asks for $150$ data points. With $19$ of them, $36$ of every $100$ validation errors land farther than $\epsilon$ from the risk.

![Figure 4 of the entry valset](https://dictionaryofml.org/terms/fig/valset-fig4.svg)

*Figure 4: Measured share of validation sets whose validation error deviates from the risk by more than the half-width $\epsilon$, when the validation set holds exactly the $\lceil \ln(2/\delta) / (2 \epsilon^{2}) \rceil$ data points that Fig. 3 requires. Each share is measured over $200{,}000$ validation sets, for a loss with values in $\{0, 1\}$ and risk $1/2$. The three confidence parameters of Fig. 3 keep their styles: $\delta = 0.01$ (solid curve), $0.1$ (dashed curve), and $0.5$ (dotted curve). The three gray segments at the right margin, labeled with their $\delta$, mark the nominal values. Every measured share stays below its $\delta$ by a factor of at least four. The vertical axis is logarithmic. Data generated by pythondemos/valset.py*

When data points are scarce, holding back a large validation set leaves few data points for the training set. As a remedy, k-fold cross-validation divides the dataset into $k$ folds. For each fold in turn, it uses that fold as validation set and the other $k - 1$ folds as training set. Each of the $k$ validation errors is therefore computed on different data points and for a different learned hypothesis. Every fold yields a noisy validation error, an average of the losses over the few data points of the fold, and averaging the $k$ per-fold validation errors produces a more reliable estimate of the risk (Hastie et al., 2009, Sect. 7.10.1; Stone, 1974).

A typical machine learning workflow therefore splits the available dataset three ways. A training set learns the hypotheses, a validation set selects among them, and a test set assesses the selected one. The test set is used only once, and the folds of k-fold cross-validation can take the place of the validation set.

**See also:** validation, [training set](https://dictionaryofml.org/terms/trainset.html), [test set](https://dictionaryofml.org/terms/testset.html), [data point](https://dictionaryofml.org/terms/datapoint.html), risk, [hypothesis](https://dictionaryofml.org/terms/hypothesis.html), [loss](https://dictionaryofml.org/terms/loss.html), [validation error](https://dictionaryofml.org/terms/valerr.html), [training error](https://dictionaryofml.org/terms/trainerr.html), [$k$-fold cross-validation](https://dictionaryofml.org/terms/kfoldcv.html), leave-one-out cross-validation, [model selection](https://dictionaryofml.org/terms/modelsel.html).

## References

- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Stone (1974). Cross-Validatory Choice and Assessment of Statistical Predictions. *Journal of the Royal Statistical Society. Series B (Methodological)*. <https://doi.org/10.1111/j.2517-6161.1974.tb00994.x>

## Cite this entry

```bibtex
@misc{dictml_valset,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {validation set},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/valset.html}
}
```


---

# variance

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/variance.html) · [typeset PDF](https://dictionaryofml.org/terms/variance.pdf) · [LaTeX](https://dictionaryofml.org/terms/variance.tex) · [Python demo](https://dictionaryofml.org/terms/variance.py) · [notebook](https://dictionaryofml.org/terms/variance.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

The variance of a real-valued random variable is the expectation of the squared difference between the random variable and its mean. It quantifies the spread of the probability distribution of the random variable around the mean: a small variance means that realizations concentrate near the mean. The term also refers to the sample variance of a dataset; the two usages are consistent, since the sample variance is the variance of the random variable obtained by drawing a data point uniformly from the dataset. Under the squared error loss, the variance is the smallest risk that a constant prediction of a numeric label can achieve. For a random vector, the variance is the expectation of the squared Euclidean norm of the deviation from the mean. This quantity equals the trace of the covariance matrix of the random vector, i.e., the sum of the variances of its entries.

## Definition

The Helsinki Kaisaniemi weather station recorded seven daily maximum temperatures between $19.6$ and $22.2\,^{\circ}$C on 24–30 August 2026 (Finnish Meteorological Institute, 2026), and every reading lies within two degrees of their average of $20.5\,^{\circ}$C. How far readings scatter around their average is what the variance measures.

The variance of a real-valued random variable $y$ is the expectation $\mathbb{E}\big\{ \big( y - \mathbb{E}\{y \} \big)^{2} \big\}$ of the squared difference between $y$ and its expectation $\mathbb{E}\{y \}$, also called its mean. The term is also used to refer to the sample variance of a finite dataset $\mathcal{D} = \big\{ y^{(1)}, \,\ldots, \,y^{(m)} \in \mathbb{R} \big\}$. These two usages are consistent. A dataset defines a discrete random variable $\widetilde{y}^{(\mathcal{D})} = y^{(I)}$ on the sample space $\{1, \,\ldots, \,m\}$. Here, the index $I$ is chosen uniformly at random, i.e., $\mathbb{P}\left(I=r\right)=1/m$ for all $r=1,\ldots,m$, so the probability distribution of $\widetilde{y}^{(\mathcal{D})}$ is the empirical distribution of the dataset. The variance of $\widetilde{y}^{(\mathcal{D})}$ is precisely the sample variance

$$
\frac{1}{m} \sum_{r=1}^{m}
\Big( y^{(r)} - \frac{1}{m}
\sum_{r'=1}^{m} y^{(r')}
\Big)^{2} \text{,}
$$

i.e., the average squared deviation of the data points from their sample mean. Fig. 1 depicts the seven Helsinki readings of 24–30 August 2026 as realizations of a real-valued random variable, together with their sample mean and the deviation of each measurement from it.

![Figure 1 of the entry variance](https://dictionaryofml.org/terms/fig/variance-fig1.svg)

*Figure 1: Daily maximum temperatures $y^{(r)}$ measured at the Helsinki Kaisaniemi weather station on 24–30 August 2026 (Finnish Meteorological Institute, 2026) (filled circles), their sample mean (dashed line), and the deviation of each measurement from it (vertical segments). The sample variance of the measurements is the average of the squared deviations. Data generated by pythondemos/variance.py*

For the seven temperatures of Fig. 1, the sample variance is the number $6.34/7 \approx 0.91$, computed from the readings alone. It estimates the variance of the random variable that generated them, which is a property of its probability distribution and is never observed exactly. The sample variance also serves as a threshold for detecting outliers: a common rule declares a data point an outlier when its squared deviation from the sample mean exceeds nine times the sample variance, i.e., three standard deviations (Pukelsheim, 1994, Sect. 1). The reading of $22.2\,^{\circ}$C deviates by $1.7\,^{\circ}$C, whose square $2.89$ is about three times the sample variance, so the rule keeps every reading.

For a random variable with a finite second moment, i.e., $\mathbb{E}\{ y^{2} \} < \infty$, the mean is the constant $b$ that minimizes the risk $\mathbb{E}\{ ( y - b )^{2} \}$. The variance is the smallest value this risk attains, i.e., its minimum at $b = \mathbb{E}\{y\}$ (Bertsekas and Tsitsiklis, 2008, Sect. 8.3):

$$
\mathbb{E}\big\{ \big( y - \mathbb{E}\{y\} \big)^{2} \big\}
= \min_{b \in \mathbb{R}}
\mathbb{E}\big\{ ( y - b )^{2} \big\} \text{.}
$$

This minimum-risk characterization applies directly to the simplest machine learning problem, predicting a numeric label $y$ without using any features. The hypothesis space for this problem consists of the constant maps $h(\cdot) = b$ with $b \in \mathbb{R}$. Each such hypothesis amounts to a single bias term: it delivers the same prediction $b$ for every data point. Given a training set $\mathcal{D}^{(\rm train)} = \big\{ y^{(1)}, \,\ldots, \,y^{(m)} \big\}$ that consists of labels $y^{(r)} \in \mathbb{R}$ only, empirical risk minimization with the squared error loss amounts to finding the constant that best predicts the label of a data point. The solution $\widehat{b}$ is the sample mean of the labels (see mean), and the training error of the learned hypothesis is the sample variance of the labels (Jung, 2022). For the temperatures of Fig. 1, the learned hypothesis is the dashed line: $\widehat{b} = 20.5\,^{\circ}$C, with training error equal to the sample variance $0.91$ computed above. The variance of the label therefore serves as a baseline: a machine learning method that uses features is useful only if its risk falls below the variance of the label.

Variance extends from a real-valued random variable to a random vector ${\mathbf x}$ as the expectation of the squared Euclidean norm of the deviation from the mean, $\mathbb{E}\big\{ \left\lVert {\mathbf x} - \mathbb{E}\{{\mathbf x} \} \right\rVert_{2}^{2} \big\} = {\rm tr} \left(\mathbf{C}^{({\mathbf x})}\right)$. The squared Euclidean norm is the sum of the squared deviations of the entries of ${\mathbf x}$, so its expectation is the sum of their variances. These are the diagonal entries of the covariance matrix $\mathbf{C}^{({\mathbf x})}$ (Murphy, 2012, Sect. 2.5.1), and the sum of the diagonal entries is the trace.

**Synonyms:** second central moment.

**See also:** [random variable](https://dictionaryofml.org/terms/rv.html), random vector, expectation, [mean](https://dictionaryofml.org/terms/mean.html), sample variance, standard deviation, [probability distribution](https://dictionaryofml.org/terms/probdist.html), [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), squared error loss, [baseline](https://dictionaryofml.org/terms/baseline.html), [covariance matrix](https://dictionaryofml.org/terms/covmtx.html), trace, outlier.

## References

- Finnish Meteorological Institute (2026). The Finnish Meteorological Institute's open data. <https://opendata.fmi.fi/wfs>
- Pukelsheim (1994). The Three Sigma Rule. *The American Statistician*.
- Bertsekas and Tsitsiklis (2008). Introduction to Probability. *Athena Scientific*.
- Jung (2022). Machine Learning: The Basics. *Springer Nature*. <https://doi.org/10.1007/978-981-16-8193-6>
- Murphy (2012). Machine Learning: A Probabilistic Perspective. *MIT Press*.

## Cite this entry

```bibtex
@misc{dictml_variance,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {variance},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/variance.html}
}
```


---

# explainable artificial intelligence (XAI)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-10-29. [HTML](https://dictionaryofml.org/terms/xaiterm.html) · [typeset PDF](https://dictionaryofml.org/terms/xaiterm.pdf) · [LaTeX](https://dictionaryofml.org/terms/xaiterm.tex) · [Python demo](https://dictionaryofml.org/terms/xaiterm.py) · [notebook](https://dictionaryofml.org/terms/xaiterm.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Explainable artificial intelligence (XAI) is the subfield of artificial intelligence concerned with making the predictions of machine learning methods understandable to humans. Much of it can be posed as a function approximation problem: a learned hypothesis is explained by a simpler function that a human can comprehend. Explanations differ in what they are made of. One kind reports a score for each feature, the weights of a linear map fitted near the data point (local interpretable model-agnostic explanations) or additive contributions to the prediction (SHapley Additive exPlanations). A counterfactual instead names another feature vector, the closest one whose prediction differs. A concept activation vector states the explanation in a concept the user names, located in the activations inside the trained method. All three are post hoc and treat the learned hypothesis as fixed. The alternative is a hypothesis that needs no separate explanation, reached by restricting the hypothesis space to hypotheses a human comprehends or by regularization that favors such hypotheses. Explainable empirical risk minimization instead keeps the hypothesis space and penalizes deviations from the predictions of one specific user.

## Definition

A bank's artificial intelligence system rejects a loan application, and the applicant asks why. XAI is the subfield of artificial intelligence concerned with making the predictions of machine learning methods understandable to humans. It aims to complement each prediction by an explanation of how it has been obtained. If the machine learning method uses a sufficiently simple hypothesis space, the learned hypothesis may be inherently interpretable and no separate explanation is needed (Rudin, 2019); see interpretable machine learning, discussed further below.

The term XAI was popularized by a program of the US Defense Advanced Research Projects Agency (Gunning and Aha, 2019); in the context of machine learning, the synonymous term explainable machine learning is also used. The underlying property is explainability, which the international terminology standard for artificial intelligence defines for AI systems in general (International Organization for Standardization and International Electrotechnical Commission, 2022, Sect. 3.5.7).

Much of XAI can be posed as a function approximation problem. A trained hypothesis $\widehat{h}: \mathcal{X} \rightarrow \mathcal{Y}$, e.g., delivered by an opaque deep net, is typically a composition of many nonlinear layers, a function on a high-dimensional feature space $\mathcal{X}$ (Goodfellow et al., 2016, Ch. 6). Explaining $\widehat{h}$ at a data point with feature vector ${\mathbf x} \in \mathcal{X}$ means answering one question about that function in a form a human can comprehend. The examples of explanations below are of three kinds, distinguished by what the explanation is made of: the first approximates $\widehat{h}$ by a simpler function, the second asks where $\widehat{h}$ takes a different value, and the third refers to a concept inside the trained method.

The first is a score for each feature of the data point. The scores are the weights of a simpler hypothesis, such as a linear map, that agrees with $\widehat{h}$ near ${\mathbf x}$ and comes from a hypothesis space whose hypotheses a user can read, as local interpretable model-agnostic explanations computes them. The second is another feature vector, one whose prediction differs. The third is a concept the user names, located inside the trained method rather than among the features of the data point. All three are post hoc: they treat the learned hypothesis as fixed and construct the explanation after training.

![Figure 1 of the entry xaiterm](https://dictionaryofml.org/terms/fig/xaiterm-fig1.svg)

*Figure 1: The first kind of explanation. The learned hypothesis $\widehat{h}$ delivers the prediction $\widehat{h}({\mathbf x})$ for a data point with feature vector ${\mathbf x}$. The explanation is constructed from a simple hypothesis $g$, e.g., a linear map, that approximates $\widehat{h}$ near ${\mathbf x}$; the score reported for a feature is the weight $g$ gives it (cf. local interpretable model-agnostic explanations)*

The first kind of explanation reports one score per feature, saying how much that feature contributed to the prediction. Local interpretable model-agnostic explanations obtains the scores as the weights of a linear map fitted to $\widehat{h}$ near ${\mathbf x}$ (Ribeiro et al., 2016); Fig. 1 draws that construction. SHapley Additive exPlanations obtains them by decomposing the prediction into a base value and one contribution per feature, the contributions being Shapley values (Lundberg and Lee, 2017; Molnar, 2025). The two deliver the same kind of explanation and differ in how its scores are computed. When features are properties of image pixels, the feature scores are a second image of the same size: one relevance score per pixel (see Fig. 2). The class activation map is one construction of such a per-pixel score.

![Figure 2 of the entry xaiterm](https://dictionaryofml.org/terms/fig/xaiterm-fig2.svg)

*Figure 2: An explanation of the first kind for an image, one score per feature, as a class activation map delivers it. The objects are named as in Fig. 1. Left: a data point ${\mathbf x}$, an image on a grid of $6 \times 6$ pixels, for which the learned hypothesis delivers the prediction $\widehat{h}({\mathbf x}) = $ *seven*. Right: the explanation of that prediction, one relevance score per pixel. The pixels of the diagonal stroke carry the largest scores, the top bar smaller ones, and the remaining pixels score zero*

The second kind of explanation is a counterfactual. It reports no scores and approximates nothing. It asks instead where $\widehat{h}$ takes a different value, pointing out how the features must be modified for the prediction $\widehat{h}({\mathbf x})$ to come out differently. A counterfactual identifies the smallest such modification, with closeness measured by a chosen metric (Wachter et al., 2018) (see Fig. 3).

![Figure 3 of the entry xaiterm](https://dictionaryofml.org/terms/fig/xaiterm-fig3.svg)

*Figure 3: The first two kinds of explanation for one prediction. The trained hypothesis $\widehat{h}$ (solid curve) delivers a prediction for each value of the feature. At a given data point with feature value $x$ (filled circle), local interpretable model-agnostic explanations (dashed line) constructs a local linear approximation. A counterfactual identifies the closest point $x'$ (open circle), with respect to a chosen metric, at which the prediction crosses a decision threshold. The double arrow marks the change of the feature it reports*

For the rejected loan applicant, for example, local interpretable model-agnostic explanations reports which of the applicant's features, such as income and credit history, its local linear approximation assigns the largest weight to in the rejection. A counterfactual instead states the smallest modification of those features that would turn the prediction into an approval.

The third kind of explanation refers not to the features of the data point but to a concept the user specifies. Such a concept is defined by a dataset of examples the user supplies. A concept activation vector locates such a concept as a direction in the space of the activations of one hidden layer of an artificial neural network: the normal vector of a decision boundary that separates the supplied examples carrying the concept from those that do not (Kim et al., 2018).

In contrast to local interpretable model-agnostic explanations and counterfactual explanations, a concept activation vector needs access to the internal computations of a machine learning system. One such internal computation is the activation ${\mathbf z} = f({\mathbf x})$ of a hidden layer in a deep net. A deep net computes $\widehat{h}$ by feeding the activations ${\mathbf z}$ to the final layers, which deliver a score $s({\mathbf z})$ for each label; the prediction $\widehat{h}({\mathbf x})$ follows from those scores (see concept activation vector). In an image classifier, for example, a user who suspects that photographs are labeled zebra through the concept *stripes* supplies photographs that show stripes and photographs that do not. The concept activation vector for *stripes* is then the normal vector of the linear classifier that separates the activations ${\mathbf z}$ of the two sets of photographs. How much the concept contributed to the prediction of a zebra photograph is then measured by the directional derivative of the score $s$ along that concept activation vector.

The above XAI methods construct an explanation after training, for a learned hypothesis that is already fixed. The alternative is to learn one that needs no separate explanation. Interpretable machine learning does so by admitting only hypotheses a human comprehends directly. This can be achieved by manually choosing a hypothesis space that is simple enough, or by regularization that favors hypotheses with specific properties, such as sparsity or predictability (Tibshirani, 1996; Zhang et al., 2024). Explainable empirical risk minimization uses regularization but not to that end: it leaves the hypothesis space as it is and delivers explainability to one specific user. That user supplies their own predictions for the data points of a training set, and the penalty term measures the deviation of the predictions of the learned hypothesis from the user predictions (Zhang et al., 2024).

Three aspects of an explanation are studied: whom it serves, how much of the learned hypothesis it covers, and whether it is faithful. How much an explanation achieves depends on who receives it: explainability is measured relative to a specific user (Colin et al., 2022; Jung and Nardelli, 2020). Explanations can be local, concerning a single prediction, or global, characterizing the learned hypothesis as a whole (Molnar, 2025).

An explanation must be faithful, i.e., reflect the computation that the learned hypothesis carries out. When the explanation scores the features, this can be tested rather than asserted. A class activation map is faithful for a prediction if flipping the pixels it scores highest changes that prediction more often than flipping as many of the pixels it scores lowest. Perturbing the highest-scoring regions first and recording how quickly the predicted class score falls is the standard form of the test (Samek et al., 2017, Sect. III-A, p. 2666).

For the image of Fig. 2 and a linear classifier fitted to images of that kind, flipping the three highest-scoring pixels changes the prediction, while flipping the pixels in the opposite order leaves it unchanged through all $36$ of them. The map-guided order also changes the prediction sooner on each of $200$ noisy variants of the image. Faithfulness also limits what a post hoc explanation can achieve: one that agreed with the learned hypothesis everywhere would be that hypothesis itself, so a simpler explanation deviates from it somewhere (Rudin, 2019).

Two legal instruments explicitly state requirements on XAI. Under the general data protection regulation, a person subjected to a decision taken by automated means is entitled to ``meaningful information about the logic involved'' in that decision (European Parliament and Council of the European Union, 2016, Art. 15(1)(h)). Communicating the algorithm itself is not a sufficiently concise and intelligible explanation (Court of Justice of the European Union, 2025). Counterfactual information can be appropriate: the extent to which a variation in the personal data would have led to a different result. The second instrument is the EU AI Act, which requires for a high-risk artificial intelligence system that affected persons obtain ``clear and meaningful explanations of the role of the AI system in the decision-making procedure'' (European Parliament and Council of the European Union, 2024, Art. 86) (see right to explanation).

**Synonyms:** explainable machine learning (explainable ML).

**See also:** [explainability](https://dictionaryofml.org/terms/explainability.html), [interpretability](https://dictionaryofml.org/terms/interpretability.html), [explanation](https://dictionaryofml.org/terms/explanation.html), [interpretable machine learning](https://dictionaryofml.org/terms/interpretableml.html), explainable empirical risk minimization, local interpretable model-agnostic explanations, SHapley Additive exPlanations, counterfactual, [feature](https://dictionaryofml.org/terms/feature.html), mechanistic interpretability, class activation map, right to explanation.

## References

- Rudin (2019). Stop explaining black box machine learning models for high-stakes decisions and use interpretable models instead. *Nature Mach. Intell.*. <https://doi.org/10.1038/s42256-019-0048-x>
- Gunning and Aha (2019). DARPA's Explainable Artificial Intelligence Program. *AI Magazine*. <https://doi.org/10.1609/aimag.v40i2.2850>
- International Organization for Standardization and International Electrotechnical Commission (2022). ISO/IEC 22989:2022 --- Information technology --- Artificial intelligence --- Artificial intelligence concepts and terminology. <https://www.iso.org/standard/74296.html>
- Goodfellow et al. (2016). Deep Learning. *MIT Press*.
- Ribeiro et al. (2016). ``Why Should I Trust You?'': Explaining the Predictions of Any Classifier. *Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining*. <https://doi.org/10.1145/2939672.2939778>
- Lundberg and Lee (2017). A Unified Approach to Interpreting Model Predictions. *Advances in Neural Information Processing Systems 30*. <https://proceedings.neurips.cc/paper_files/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html>
- Molnar (2025). Interpretable Machine Learning: A Guide for Making Black Box Models Explainable. *Ebook*. <https://christophm.github.io/interpretable-ml-book/>
- Wachter et al. (2018). Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR. *Harvard Journal of Law & Technology*. <https://doi.org/10.2139/ssrn.3063289>
- Kim et al. (2018). Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). *Proc. 35th Int. Conf. Mach. Learn.*. <https://proceedings.mlr.press/v80/kim18d.html>
- Tibshirani (1996). Regression Shrinkage and Selection via the Lasso. *J. Roy. Statist. Soc.: Ser. B (Methodological)*. <https://doi.org/10.1111/j.2517-6161.1996.tb02080.x>
- Zhang et al. (2024). Explainable empirical risk minimization. *Neural Comput. Appl.*. <https://doi.org/10.1007/s00521-023-09269-3>
- Colin et al. (2022). What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods. *Adv. Neural Inf. Process. Syst.*. <https://proceedings.neurips.cc/paper_files/paper/2022/hash/13113e938f2957891c0c5e8df811dd01-Abstract-Conference.html>
- Jung and Nardelli (2020). An Information-Theoretic Approach to Personalized Explainable Machine Learning. *IEEE Signal Process. Lett.*. <https://doi.org/10.1109/lsp.2020.2993176>
- Samek et al. (2017). Evaluating the Visualization of What a Deep Neural Network Has Learned. *IEEE Trans. Neural Netw. Learn. Syst.*. <https://doi.org/10.1109/TNNLS.2016.2599820>
- European Parliament and Council of the European Union (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (Text with EEA relevance). <https://eur-lex.europa.eu/eli/reg/2016/679/oj>
- Court of Justice of the European Union (2025). Judgment of 27 February 2025, CK v Dun & Bradstreet Austria GmbH and Magistrat der Stadt Wien, C-203/22, ECLI:EU:C:2025:117. <https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:62022CJ0203>
- European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act) (Text with EEA relevance). <https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng>

## Cite this entry

```bibtex
@misc{dictml_xaiterm,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {explainable artificial intelligence (XAI)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/xaiterm.html}
}
```