# training error

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-10-29. [HTML](https://dictionaryofml.org/terms/trainerr.html) · [typeset PDF](https://dictionaryofml.org/terms/trainerr.pdf) · [LaTeX](https://dictionaryofml.org/terms/trainerr.tex) · [Python demo](https://dictionaryofml.org/terms/trainerr.py) · [notebook](https://dictionaryofml.org/terms/trainerr.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

The training error of a hypothesis is its average loss over the training set, the empirical risk of the hypothesis on the data points used for training. For a hypothesis delivered by empirical risk minimization, it is the smallest empirical risk that any hypothesis of the hypothesis space attains. Because the same data points selected the hypothesis, the training error is often far below the risk, by a gap that grows with the size of the hypothesis space, so a hypothesis with zero training error can have a large risk. Under the independent and identically distributed assumption, the training error of every hypothesis is a noisy estimate of its risk, and empirical risk minimization compares all of them at once, so a generalization guarantee for the empirical risk minimization solution requires the deviation to be small uniformly over the hypothesis space. A training error far below the validation error indicates overfitting, and a large training error indicates underfitting.

## Definition

Twenty of the $1827$ days of five years, 2020 to 2024, at the weather station Krems an der Donau (Austria), every $92$nd day, are drawn in Fig. 1 as a scatterplot. Each day is a data point: its feature $x$ is the morning minimum temperature, its label $y$ the maximum temperature of the day. Three of the days, drawn in black, form a training set, the days used for training, for two empirical risk minimization-based methods that deliver the hypotheses $\widehat{h}^{(1)}$ and $\widehat{h}^{(2)}$. The hypothesis $\widehat{h}^{(1)}$ is learned by empirical risk minimization with the squared error loss over the hypothesis space of affine maps $h(x) = b + w x$, the linear model. The dotted bars are its prediction errors on the three days, the differences between the recorded maximum and the value of $\widehat{h}^{(1)}$, and the average of the three squared bars is its training error, $7.91$.

The second hypothesis $\widehat{h}^{(2)}$ is learned by empirical risk minimization over the hypothesis space of all polynomials of degree at most two. Although the training error of $\widehat{h}^{(2)}$ is zero, and therefore far below that of $\widehat{h}^{(1)}$, it is the worse choice: averaged over all days of the five years, the loss of $\widehat{h}^{(2)}$ is $42.90$ against $24.22$ for $\widehat{h}^{(1)}$.

![Figure 1 of the entry trainerr](https://dictionaryofml.org/terms/fig/trainerr-fig1.svg)

*Figure 1: Twenty of the $1827$ days at Krems an der Donau, 2020 to 2024 (every $92$nd day), each a gray dot with the morning minimum temperature as its feature and the maximum temperature of the day as its label. The three black dots form the training set. The solid line is the hypothesis $\widehat{h}^{(1)}$ that empirical risk minimization learns from the three days over the linear model, and the dotted bars are its prediction errors on them; the average of the three squared bars is its training error. The dashed curve is the degree-two polynomial $\widehat{h}^{(2)}$ that empirical risk minimization learns over the polynomials of degree at most two, whose training error is zero; left of the coldest training day it falls below the frame. Data generated by pythondemos/trainerr.py*

Holding the intercept of $\widehat{h}^{(1)}$ at the value empirical risk minimization gave it and varying only its slope $w$ leaves the training error a function of one number. Fig. 2 shows it with the risk, the expected loss over the probability distribution of a day, approximated by the average over the five years. The two functions have different minimizers: the training error is smallest at the slope that empirical risk minimization picked, the risk at a smaller slope. At the slope empirical risk minimization picked, the training error is a third of the risk. The training error is an average over the three days that determined the hypothesis; it does not measure the loss on a day outside the training set.

![Figure 2 of the entry trainerr](https://dictionaryofml.org/terms/fig/trainerr-fig2.svg)

*Figure 2: The training error (solid curve) and the risk (dashed curve) of the hypothesis $h^{(w)}(x) = 10.21 + w x$ as functions of its slope $w$, for the three training days and the five years of Fig. 1; the intercept is the one empirical risk minimization gave $\widehat{h}^{(1)}$. The training error is smallest at the slope that empirical risk minimization picks (filled circle), where it is far below the risk; the risk is smallest at a smaller slope (open square). Data generated by pythondemos/trainerr.py*

The training error $E_{t}$ of a hypothesis $\widehat{h}$ is its average loss over the training set $\mathcal{D}^{(\rm train)} = \{{\mathbf z}^{(r)}\}_{r=1}^{m}$,

$$
E_{t} = \frac{1}{m} \sum_{r=1}^{m}
L\left({\mathbf z}^{(r)},\widehat{h} \right) \text{,}
$$

i.e., the empirical risk of $\widehat{h}$ on the data points used for learning $\widehat{h}$. The hypothesis is the output of a training method, a map $\mathcal{A}$ that reads in a training set and delivers $\widehat{h} = \mathcal{A}(\mathcal{D}^{(\rm train)})$. For a hypothesis delivered by empirical risk minimization, the training error is the smallest empirical risk that any hypothesis of the hypothesis space attains on $\mathcal{D}^{(\rm train)}$, since empirical risk minimization minimizes this average.

The training error is computed on the same data points that selected $\widehat{h}$, which limits its use as a measure of quality. The goal of a machine learning method is to learn a hypothesis that incurs a small loss on any data point, also those outside the training set, that is, to minimize the risk introduced above. The training error of the learned hypothesis is often a poor estimator for its risk, because the learned hypothesis has been adapted to the training set by minimizing the training error. A hypothesis with zero training error can therefore have a large risk, as $\widehat{h}^{(2)}$ of Fig. 1 does. The difference between the risk and the training error is the generalization gap.

Under the independent and identically distributed assumption, the training error becomes a random variable, as it depends on the randomly generated training set. For one fixed hypothesis $h \in \mathcal{H}$, the empirical risk $\widehat{L}\big(h|\mathcal{D}^{(\rm train)}\big)$ is an average of the $m$ loss values $L\left({\mathbf z}^{(1)},h \right), \ldots,
L\left({\mathbf z}^{(m)},h \right)$, which are realizations of independent and identically distributed RVs. It is therefore the realization of a random variable with expectation $\bar{L} \big( h \big)$ and a variance proportional to $1/m$. Empirical risk minimization does not evaluate this estimate at one fixed hypothesis: it compares the estimates of all hypotheses in $\mathcal{H}$ and picks the smallest. The estimates form the family $\{ \widehat{L}\big(h|\mathcal{D}^{(\rm train)}\big) \}_{h \in \mathcal{H}}$, a stochastic process indexed by the hypothesis space: one random variable for each hypothesis, all realized by the single draw of $\mathcal{D}^{(\rm train)}$. The minimizer $\widehat{h}$ of this process is itself a random variable, and its training error $E_{t} = \widehat{L}\big(\widehat{h}|\mathcal{D}^{(\rm train)}\big)$ is the minimum of many noisy estimates, so it is below $\bar{L} \big( \widehat{h} \big)$ on average.

A guarantee of generalization for the empirical risk minimization solution must therefore hold uniformly over the hypothesis space: if

\begin{equation}
\tag{1}
\sup_{h \in \mathcal{H}}
\big| \widehat{L}\big(h|\mathcal{D}^{(\rm train)}\big) - \bar{L} \big( h \big) \big|
\leq \epsilon \text{,}
\end{equation}

then $\bar{L} \big( \widehat{h} \big) \leq E_{t} + \epsilon$, and the risk of $\widehat{h}$ exceeds the smallest risk in $\mathcal{H}$ by at most $2\epsilon$ (Shalev-Shwartz and Ben-David, 2014, Lemma 4.2). A hypothesis space for which the uniform deviation Eq. (1) falls below any $\epsilon$ with probability at least $1-\delta$ once $m$ is large enough has the uniform convergence property (Shalev-Shwartz and Ben-David, 2014, Def. 4.3). For a finite hypothesis space and a loss with values in $[0,1]$, a union bound over the hypotheses and Hoeffding's inequality give $\epsilon = \sqrt{\ln(2 |\mathcal{H}| / \delta) / (2 m)}$ (Shalev-Shwartz and Ben-David, 2014, Cor. 4.6). For an infinite hypothesis space, $\ln |\mathcal{H}|$ is replaced by a measure of its size such as the Vapnik–Chervonenkis dimension or the Rademacher complexity (Shalev-Shwartz and Ben-David, 2014, Thm. 6.8, Thm. 26.5).

In contrast, the validation error, computed as the average loss on data points not used in the training, is often a much better estimator for the risk of the learned hypothesis. For example, let the data points of the validation set be realizations of independent and identically distributed RVs, independent of the training set. Each summand of the validation error is then the loss on an independent draw from the probability distribution of a data point, whose expectation conditional on the training set is the risk of the learned hypothesis (Hastie et al., 2009, Eq. (7.2)). The expectation of the validation error conditional on the training set is therefore that risk. Moreover, the validation error estimates the risk of the single hypothesis that $\mathcal{A}$ delivered, so one Hoeffding bound without $|\mathcal{H}|$ suffices for it, and it concentrates around the risk at a rate that depends only on the size of the validation set, not on how the hypothesis was learned (Shalev-Shwartz and Ben-David, 2014, Thm. 11.1).

Fig. 3 shows three deviations of a training error from a risk for one year of the Krems record: for one fixed hypothesis, for the hypothesis that empirical risk minimization learns, and the largest deviation over the hypothesis space. The $366$ days of the year stand in for the probability distribution of a day, so the risk of a hypothesis is its average loss over the year, and a training set of $m$ days is drawn at random from the year, $300$ times for each $m \in \{10, 20, 40, 80\}$. The hypothesis space is the linear model, which consists of all hypothesis maps $h(x) = w_{1} x + w_{0}$ with $w_{1}, w_{0} \in \mathbb{R}$.

The fixed hypothesis is the one with the smallest risk in the hypothesis space, the hypothesis learned by linear regression from all $366$ days; it does not depend on the draw. The empirical risk minimization hypothesis is learned by linear regression from the drawn training set. The largest deviation over the hypothesis space, the uniform deviation Eq. (1), is approximated by the maximum over a grid of $2500$ hypotheses around the learned one. Over the repeated draws, the deviation of the fixed hypothesis has mean zero. The training error of the empirical risk minimization hypothesis is below its risk on average, and the largest deviation is several times the standard deviation of the fixed hypothesis' deviation. The gap of the empirical risk minimization hypothesis never exceeds the largest deviation, as the uniform deviation Eq. (1) requires, and both decrease as the training set grows (pythondemos/trainerr.py).

![Figure 3 of the entry trainerr](https://dictionaryofml.org/terms/fig/trainerr-fig3.svg)

*Figure 3: Deviation of the training error from the risk over repeated random draws of a training set of fixed size from the days of 2024 at Krems an der Donau, mean and one standard deviation either side. For one fixed hypothesis (filled circles) the deviation has mean zero. For the hypothesis that empirical risk minimization learns from the draw (filled squares) it is negative on average. The largest deviation over the hypothesis space (open triangles), the uniform deviation Eq. (1) approximated on a grid of $2500$ hypotheses, bounds the empirical risk minimization gap in every draw and decreases with the size of the training set. Data generated by pythondemos/trainerr.py*

The size of the gap $\bar{L} \big( \widehat{h} \big) - E_{t}$ depends on the hypothesis space. The first day of each month from January to August 2024 forms a training set of eight days, and the rest of the year is held back. A hypothesis of the linear model learned from the eight days has a training error of $20.36$, only slightly below its average squared error of $21.60$ on the held-back days: the linear model has two model parameters and can adapt little to the particular eight days. Fig. 4 shows the two errors for polynomials of prescribed maximum degree $p = 0, \ldots, 7$. Increasing the degree means a larger hypothesis space, a polynomial of degree $p$ being one of degree $p+1$ with a zero leading coefficient, so the training error never increases with the degree. In contrast, the average loss on the held-back days is smallest at degree one and grows for larger degrees, where the polynomial oscillates between the eight days it was learned from.

![Figure 4 of the entry trainerr](https://dictionaryofml.org/terms/fig/trainerr-fig4.svg)

*Figure 4: Training error (solid line, filled circles) and held-back error (dashed line, filled squares) of a polynomial $\widehat{h}(x)$ of prescribed maximum degree learned by empirical risk minimization from eight days of 2024 at Krems an der Donau. The training error never increases with the degree, since increasing the degree means a larger hypothesis space. In contrast, the error on the days outside the training set is smallest at a low degree and grows for larger degrees. Data generated by pythondemos/trainerr.py*

By comparing the training error of a machine learning method with the resulting validation error, two failures can be detected. A training error far below the validation error indicates overfitting (Bishop, 2006, Sect. 1.1). A large training error indicates underfitting: a hypothesis space too small, or an iterative optimizer that stopped too early, which restricts the effective hypothesis space. Fig. 4 also shows the effect of regularization by model pruning: restricting the hypothesis space by using a small polynomial degree raises the training error a little and lowers the average loss on the held-back days.

**See also:** [training set](https://dictionaryofml.org/terms/trainset.html), [validation set](https://dictionaryofml.org/terms/valset.html), empirical risk, [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), [loss](https://dictionaryofml.org/terms/loss.html), [validation error](https://dictionaryofml.org/terms/valerr.html), risk, generalization gap, [overfitting](https://dictionaryofml.org/terms/overfitting.html), underfitting, [hypothesis](https://dictionaryofml.org/terms/hypothesis.html), [independent and identically distributed assumption](https://dictionaryofml.org/terms/iidasspt.html), [generalization](https://dictionaryofml.org/terms/generalization.html), stochastic process, concentration inequality, Vapnik–Chervonenkis dimension.

## References

- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>

## Cite this entry

```bibtex
@misc{dictml_trainerr,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {training error},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/trainerr.html}
}
```