Dictionary of Applied Machine Learning: 115 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

validation error

Updated on 2026-11-11

Readiness 100/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Markdown Cite this entry

See also validation set training set test set training error loss hypothesis empirical risk minimization overfitting model selection independent and identically distributed assumption

The validation error $E_{v}$ of a learned hypothesis is its average loss over a validation set. For a fixed hypothesis and data points that are realizations of independent and identically distributed (i.i.d.) random variables (RVs), the validation error is an unbiased estimator of the risk. If the training set is itself modelled as random, the learned hypothesis becomes an RV too, and the validation error is unbiased conditioned on the learned hypothesis. The validation error characterizes a learned hypothesis that some machine learning (ML) method obtained from a training set. It can therefore be used to choose between different models or, more generally, between different ML methods. The validation error also helps to diagnose an ML method and to guide its revision.

Figure produced by the Python demo of the entry validation error
Figure written by the entry's Python demo valerr.py, which recomputes what the entry states: the script block by block · open in Colab

Definition

B-openingThe weather station Krems an der Donau (Austria) recorded the morning minimum and the maximum temperature of every day from 2020 to 2024. Each day is a data point: its feature $x$ is the morning minimum temperature, its label $y$ the maximum temperature of the day. Fig. 1 shows twenty of them as a scatterplot and singles out six. Three of the days, drawn in black, form the training set. The hypothesis $\widehat{h}^{(2)}$ learned from them is a degree-two polynomial through all three days, so its training error is zero. Three further days, drawn as red triangles, form a validation set. The dotted bars are the prediction errors of the polynomial on them, the differences between the recorded maximum and the value of the curve. The average of the three squared bars is the validation error of the polynomial, $42.58$.

Figure 1 of the entry valerr
Figure 1: Twenty of the $1827$ days at Krems an der Donau, 2020 to 2024 (every $92$nd day), each a gray dot with the morning minimum temperature as its feature $\feature$ and the maximum temperature of the day as its label $\truelabel$. The three black dots form the training set, and the dashed curve is the degree-two polynomial $\learnthypothesis^{(2)}$ learned from them; it passes through all three, so its training error is zero. The three red triangles indicate days that form the validation set, and the dotted bars are the prediction errors of the polynomial on them. The average of the three squared bars is the validation error. Data generated by pythondemos/valerr.py
The validation error is the average loss of $\widehat{h}^{(2)}$ on days that did not belong to the training set. The training error carries no information about such days: the polynomial passes through the three training days by construction. The validation error is close to the average squared error loss of $\widehat{h}^{(2)}$ over all days of the five years, $42.90$. A hypothesis with a larger training error can be the better one: the linear hypothesis that empirical risk minimization (ERM) delivers from the same three days over the linear model has a training error of $7.91$ and a validation error of $6.98$.

The validation error $E_{v}$ of a hypothesis $\widehat{h}$ is its average loss over a validation set $\mathcal{D}^{(\rm val)}$, \[ E_{v} = \frac{1}{|\mathcal{D}^{(\rm val)}|} \sum_{{\mathbf z} \in \mathcal{D}^{(\rm val)}} L\left({\mathbf z},\widehat{h} \right) \text{.} \] Because no data point of $\mathcal{D}^{(\rm val)}$ entered the training, $E_{v}$ estimates the risk of $\widehat{h}$. The training error, computed over the training set whose average loss ERM minimized, is below it on average: the same data points both selected $\widehat{h}$ and assessed it (Hastie et al., 2009, Sect. 7.4).

The validation error is an average of $|\mathcal{D}^{(\rm val)}|$ losses. For data points that are realizations of independent and identically distributed (i.i.d.) random variables (RVs), $E_{v}$ is the realization of an RV whose expectation is the risk of $\widehat{h}$, since each summand has that expectation. The validation error is therefore an unbiased estimator of that risk. Its variance is the variance of a single loss divided by $|\mathcal{D}^{(\rm val)}|$, so its standard deviation decreases as $|\mathcal{D}^{(\rm val)}|$ grows.

In practice $\widehat{h} = \mathcal{A}(\mathcal{D}^{(\rm train)})$ is the output of a training method, a map $\mathcal{A}$ that reads in the training set $\mathcal{D}^{(\rm train)}$, so it is itself the realization of an RV: a different realization of $\mathcal{D}^{(\rm train)}$ would have delivered a different hypothesis. The validation error is then unbiased conditioned on the learned hypothesis. If the data points of $\mathcal{D}^{(\rm val)}$ are independent of those of $\mathcal{D}^{(\rm train)}$, and hence of $\widehat{h}$ (Shalev-Shwartz and Ben-David, 2014, Sect. 11.2.1), the conditional expectation of $E_{v}$ given $\widehat{h}$ is the risk of that hypothesis, \[ \mathbb{E} \big\{ E_{v} \mid \widehat{h} \big\} = \bar{L} \big( \widehat{h} \big) \text{.} \] The training error has no such property: conditioned on $\widehat{h}$, the data points of $\mathcal{D}^{(\rm train)}$ are no longer i.i.d. realizations of the probability distribution of a data point, because $\widehat{h}$ was chosen to make its average loss on them small.

A guarantee for the validation error concerns one hypothesis only: for a loss with values in $[0,1]$, Hoeffding's inequality gives $|E_{v} - \bar{L} \big( \widehat{h} \big)| \leq \sqrt{\ln(2/\delta) / (2 |\mathcal{D}^{(\rm val)}|)}$ with probability at least $1-\delta$, with no term for the hypothesis space $\mathcal{H}$ (Shalev-Shwartz and Ben-David, 2014, Thm. 11.1). A guarantee for the training error of the ERM solution must hold uniformly over $\mathcal{H}$, since ERM compares all its hypotheses (Shalev-Shwartz and Ben-David, 2014, Sects. 4.1 and 11.2.1).

A numerical experiment shows how much the validation error varies when the validation set is chosen at random. Consider the hypothesis \[ \widehat{h}(x) = 10.47 + 1.05 \, x \text{,} \] obtained by linear regression from forty data points of the same station record, every ninth day of 2024. The data points of $306$ further days of that year, none of them used for training, are collected in a dataset, the test set $\mathcal{D}^{(\rm test)}$. Here $\mathcal{D}^{(\rm test)}$ serves only as the pool from which each validation set is drawn at random; its own role, assessing a selected hypothesis, is taken up at the end.

B-condFig. 2 shows the value range of the validation error of $\widehat{h}$ over repeated draws of a validation set of prescribed size from $\mathcal{D}^{(\rm test)}$. The standard deviation of the validation error decreases with increasing size of the validation set. The decrease follows the $1/\sqrt{|\mathcal{D}^{(\rm val)}|}$ form of the bound from Hoeffding's inequality above, though the bound itself does not apply, since the squared error loss is not confined to $[0,1]$.

Figure 2 of the entry valerr
Figure 2: Spread of the validation error computed for the hypothesis $\learnthypothesis(\feature) = 10.47 + 1.05 \, \feature$. The validation errors are computed for different validation sets that are, for a given size, randomly selected from $\testset$. The dataset $\testset$ consists of data points that have not been used to learn $\learnthypothesis$. Each bar spans one standard deviation either side of the mean. The dashed curve is the average loss computed over all of $\testset$. The standard deviation decreases as $|\valset|$ grows. Data generated by pythondemos/valerr.py
Beyond estimating the risk of one hypothesis, the validation error is compared with the training error and across models. A training error far below the validation error indicates overfitting (Hastie et al., 2009, Sect. 7.2). Comparing the validation errors of several candidate models is model selection (Bishop, 2006, Sect. 1.3). A model has no validation error of its own. The phrase is short for the validation error of the hypothesis that $\mathcal{A}$ delivers from that model, since $E_{v}$ is the average loss of a single hypothesis. The selected model is the one whose hypothesis attains the smallest of the compared validation errors, so that validation error is biased downward as an estimate of the risk, and a test set is kept apart for the final assessment (Bishop, 2006, Sect. 1.3; Hastie et al., 2009, Sect. 7.2).

See also: validation set, validation, training set, test set, training error, loss, risk, hypothesis, empirical risk minimization, overfitting, model selection, generalization gap, conditional expectation, independent and identically distributed assumption, unbiased estimator.

References

  1. Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media. doi.org/10.1007/978-0-387-84858-7
  2. Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. Cambridge Univ. Press. doi.org/10.1017/cbo9781107298019
  3. Bishop (2006). Pattern Recognition and Machine Learning. Springer Science+Business Media. doi.org/10.1007/978-0-387-45528-0

Cite this entry

@misc{dictml_valerr,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {validation error},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-13},
  url = {https://dictionaryofml.org/terms/valerr.html}
}