training error

Jung, Alexander

Dictionary of Applied Machine Learning: 115 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

training error

Updated on 2026-10-29

Readiness 100/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Markdown Cite this entry

The training error of a hypothesis is its average loss over the training set, the empirical risk of the hypothesis on the data points used for training. For a hypothesis delivered by empirical risk minimization (ERM), it is the smallest empirical risk that any hypothesis of the hypothesis space attains. Because the same data points selected the hypothesis, the training error is often far below the risk, by a gap that grows with the size of the hypothesis space, so a hypothesis with zero training error can have a large risk. Under the independent and identically distributed assumption (i.i.d. assumption), the training error of every hypothesis is a noisy estimate of its risk, and ERM compares all of them at once, so a generalization guarantee for the ERM solution requires the deviation to be small uniformly over the hypothesis space. A training error far below the validation error indicates overfitting, and a large training error indicates underfitting.

Figure produced by the Python demo of the entry training error — Figure written by the entry's Python demo `trainerr.py`, which recomputes what the entry states: the script block by block · open in Colab

Definition

B-fetchTwenty of the $1827$ days of five years, 2020 to 2024, at the weather station Krems an der Donau (Austria), every $92$nd day, are drawn in Fig. 1 as a scatterplot. Each day is a data point: its feature $x$ is the morning minimum temperature, its label $y$ the maximum temperature of the day. Three of the days, drawn in black, form a training set, the days used for training, for two empirical risk minimization (ERM)-based methods that deliver the hypotheses $\widehat{h}^{(1)}$ and $\widehat{h}^{(2)}$. The hypothesis $\widehat{h}^{(1)}$ is learned by ERM with the squared error loss over the hypothesis space of affine maps $h(x) = b + w x$, the linear model. The dotted bars are its prediction errors on the three days, the differences between the recorded maximum and the value of $\widehat{h}^{(1)}$, and the average of the three squared bars is its training error, $7.91$.

B-openingThe second hypothesis $\widehat{h}^{(2)}$ is learned by ERM over the hypothesis space of all polynomials of degree at most two. Although the training error of $\widehat{h}^{(2)}$ is zero, and therefore far below that of $\widehat{h}^{(1)}$, it is the worse choice: averaged over all days of the five years, the loss of $\widehat{h}^{(2)}$ is $42.90$ against $24.22$ for $\widehat{h}^{(1)}$.

Figure 1 of the entry trainerr — Figure 1: Twenty of the $1827$ days at Krems an der Donau, 2020 to 2024 (every $92$nd day), each a gray dot with the morning minimum temperature as its feature and the maximum temperature of the day as its label. The three black dots form the training set. The solid line is the hypothesis $\learnthypothesis^{(1)}$ that ERM learns from the three days over the linear model, and the dotted bars are its prediction errors on them; the average of the three squared bars is its training error. The dashed curve is the degree-two polynomial $\learnthypothesis^{(2)}$ that ERM learns over the polynomials of degree at most two, whose training error is zero; left of the coldest training day it falls below the frame. Data generated by `pythondemos/trainerr.py`

Holding the intercept of $\widehat{h}^{(1)}$ at the value ERM gave it and varying only its slope $w$ leaves the training error a function of one number. Fig. 2 shows it with the risk, the expected loss over the probability distribution of a day, approximated by the average over the five years. The two functions have different minimizers: the training error is smallest at the slope that ERM picked, the risk at a smaller slope. At the slope ERM picked, the training error is a third of the risk. The training error is an average over the three days that determined the hypothesis; it does not measure the loss on a day outside the training set.

Figure 2 of the entry trainerr — Figure 2: The training error (solid curve) and the risk (dashed curve) of the hypothesis $\hypothesis^{(\weight)}(\feature) = 10.21 + \weight \feature$ as functions of its slope $\weight$, for the three training days and the five years of Fig. 1; the intercept is the one ERM gave $\learnthypothesis^{(1)}$. The training error is smallest at the slope that ERM picks (filled circle), where it is far below the risk; the risk is smallest at a smaller slope (open square). Data generated by `pythondemos/trainerr.py`

The training error $E_{t}$ of a hypothesis $\widehat{h}$ is its average loss over the training set $\mathcal{D}^{(\rm train)} = \{{\mathbf z}^{(r)}\}_{r=1}^{m}$, \[ E_{t} = \frac{1}{m} \sum_{r=1}^{m} L\left({\mathbf z}^{(r)},\widehat{h} \right) \text{,} \] i.e., the empirical risk of $\widehat{h}$ on the data points used for learning $\widehat{h}$. The hypothesis is the output of a training method, a map $\mathcal{A}$ that reads in a training set and delivers $\widehat{h} = \mathcal{A}(\mathcal{D}^{(\rm train)})$. For a hypothesis delivered by ERM, the training error is the smallest empirical risk that any hypothesis of the hypothesis space attains on $\mathcal{D}^{(\rm train)}$, since ERM minimizes this average.

The training error is computed on the same data points that selected $\widehat{h}$, which limits its use as a measure of quality. The goal of a machine learning (ML) method is to learn a hypothesis that incurs a small loss on any data point, also those outside the training set, that is, to minimize the risk introduced above. The training error of the learned hypothesis is often a poor estimator for its risk, because the learned hypothesis has been adapted to the training set by minimizing the training error. A hypothesis with zero training error can therefore have a large risk, as $\widehat{h}^{(2)}$ of Fig. 1 does. The difference between the risk and the training error is the generalization gap.

Under the independent and identically distributed assumption (i.i.d. assumption), the training error becomes a random variable (RV), as it depends on the randomly generated training set. For one fixed hypothesis $h \in \mathcal{H}$, the empirical risk $\widehat{L}\big(h|\mathcal{D}^{(\rm train)}\big)$ is an average of the $m$ loss values $L\left({\mathbf z}^{(1)},h \right), \ldots, L\left({\mathbf z}^{(m)},h \right)$, which are realizations of independent and identically distributed (i.i.d.) RVs. It is therefore the realization of an RV with expectation $\bar{L} \big( h \big)$ and a variance proportional to $1/m$. ERM does not evaluate this estimate at one fixed hypothesis: it compares the estimates of all hypotheses in $\mathcal{H}$ and picks the smallest. The estimates form the family $\{ \widehat{L}\big(h|\mathcal{D}^{(\rm train)}\big) \}_{h \in \mathcal{H}}$, a stochastic process indexed by the hypothesis space: one RV for each hypothesis, all realized by the single draw of $\mathcal{D}^{(\rm train)}$. The minimizer $\widehat{h}$ of this process is itself an RV, and its training error $E_{t} = \widehat{L}\big(\widehat{h}|\mathcal{D}^{(\rm train)}\big)$ is the minimum of many noisy estimates, so it is below $\bar{L} \big( \widehat{h} \big)$ on average.

A guarantee of generalization for the ERM solution must therefore hold uniformly over the hypothesis space: if \begin{equation} \label{eq_trainerr_uniform} \sup_{h \in \mathcal{H}} \big| \widehat{L}\big(h|\mathcal{D}^{(\rm train)}\big) - \bar{L} \big( h \big) \big| \leq \epsilon \text{,} \end{equation} then $\bar{L} \big( \widehat{h} \big) \leq E_{t} + \epsilon$, and the risk of $\widehat{h}$ exceeds the smallest risk in $\mathcal{H}$ by at most $2\epsilon$ (Shalev-Shwartz and Ben-David, 2014, Lemma 4.2). A hypothesis space for which the uniform deviation $\eqref{eq_trainerr_uniform}$ falls below any $\epsilon$ with probability at least $1-\delta$ once $m$ is large enough has the uniform convergence property (Shalev-Shwartz and Ben-David, 2014, Def. 4.3). For a finite hypothesis space and a loss with values in $[0,1]$, a union bound over the hypotheses and Hoeffding's inequality give $\epsilon = \sqrt{\ln(2 |\mathcal{H}| / \delta) / (2 m)}$ (Shalev-Shwartz and Ben-David, 2014, Cor. 4.6). For an infinite hypothesis space, $\ln |\mathcal{H}|$ is replaced by a measure of its size such as the Vapnik–Chervonenkis dimension (VC dimension) or the Rademacher complexity (Shalev-Shwartz and Ben-David, 2014, Thm. 6.8, Thm. 26.5).

In contrast, the validation error, computed as the average loss on data points not used in the training, is often a much better estimator for the risk of the learned hypothesis. For example, let the data points of the validation set be realizations of i.i.d. RVs, independent of the training set. Each summand of the validation error is then the loss on an independent draw from the probability distribution of a data point, whose expectation conditional on the training set is the risk of the learned hypothesis (Hastie et al., 2009, Eq. (7.2)). The expectation of the validation error conditional on the training set is therefore that risk. Moreover, the validation error estimates the risk of the single hypothesis that $\mathcal{A}$ delivered, so one Hoeffding bound without $|\mathcal{H}|$ suffices for it, and it concentrates around the risk at a rate that depends only on the size of the validation set, not on how the hypothesis was learned (Shalev-Shwartz and Ben-David, 2014, Thm. 11.1).

Fig. 3 shows three deviations of a training error from a risk for one year of the Krems record: for one fixed hypothesis, for the hypothesis that ERM learns, and the largest deviation over the hypothesis space. The $366$ days of the year stand in for the probability distribution of a day, so the risk of a hypothesis is its average loss over the year, and a training set of $m$ days is drawn at random from the year, $300$ times for each $m \in \{10, 20, 40, 80\}$. The hypothesis space is the linear model, which consists of all hypothesis maps $h(x) = w_{1} x + w_{0}$ with $w_{1}, w_{0} \in \mathbb{R}$.

B-uniformThe fixed hypothesis is the one with the smallest risk in the hypothesis space, the hypothesis learned by linear regression from all $366$ days; it does not depend on the draw. The ERM hypothesis is learned by linear regression from the drawn training set. The largest deviation over the hypothesis space, the uniform deviation $\eqref{eq_trainerr_uniform}$, is approximated by the maximum over a grid of $2500$ hypotheses around the learned one. Over the repeated draws, the deviation of the fixed hypothesis has mean zero. The training error of the ERM hypothesis is below its risk on average, and the largest deviation is several times the standard deviation of the fixed hypothesis' deviation. The gap of the ERM hypothesis never exceeds the largest deviation, as the uniform deviation $\eqref{eq_trainerr_uniform}$ requires, and both decrease as the training set grows (pythondemos/trainerr.py).

Figure 3 of the entry trainerr — Figure 3: Deviation of the training error from the risk over repeated random draws of a training set of fixed size from the days of 2024 at Krems an der Donau, mean and one standard deviation either side. For one fixed hypothesis (filled circles) the deviation has mean zero. For the hypothesis that ERM learns from the draw (filled squares) it is negative on average. The largest deviation over the hypothesis space (open triangles), the uniform deviation $\eqref{eq_trainerr_uniform}$ approximated on a grid of $2500$ hypotheses, bounds the ERM gap in every draw and decreases with the size of the training set. Data generated by `pythondemos/trainerr.py`

The size of the gap $\bar{L} \big( \widehat{h} \big) - E_{t}$ depends on the hypothesis space. The first day of each month from January to August 2024 forms a training set of eight days, and the rest of the year is held back. A hypothesis of the linear model learned from the eight days has a training error of $20.36$, only slightly below its average squared error of $21.60$ on the held-back days: the linear model has two model parameters and can adapt little to the particular eight days. Fig. 4 shows the two errors for polynomials of prescribed maximum degree $p = 0, \ldots, 7$. Increasing the degree means a larger hypothesis space, a polynomial of degree $p$ being one of degree $p+1$ with a zero leading coefficient, so the training error never increases with the degree. In contrast, the average loss on the held-back days is smallest at degree one and grows for larger degrees, where the polynomial oscillates between the eight days it was learned from.

Figure 4 of the entry trainerr — Figure 4: Training error (solid line, filled circles) and held-back error (dashed line, filled squares) of a polynomial $\learnthypothesis(\feature)$ of prescribed maximum degree learned by ERM from eight days of 2024 at Krems an der Donau. The training error never increases with the degree, since increasing the degree means a larger hypothesis space. In contrast, the error on the days outside the training set is smallest at a low degree and grows for larger degrees. Data generated by `pythondemos/trainerr.py`

By comparing the training error of an ML method with the resulting validation error, two failures can be detected. A training error far below the validation error indicates overfitting (Bishop, 2006, Sect. 1.1). A large training error indicates underfitting: a hypothesis space too small, or an iterative optimizer that stopped too early, which restricts the effective hypothesis space. Fig. 4 also shows the effect of regularization by model pruning: restricting the hypothesis space by using a small polynomial degree raises the training error a little and lowers the average loss on the held-back days.

See also: training set, validation set, empirical risk, empirical risk minimization, loss, validation error, risk, generalization gap, overfitting, underfitting, hypothesis, independent and identically distributed assumption, generalization, stochastic process, concentration inequality, Vapnik–Chervonenkis dimension.

References

Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. Cambridge Univ. Press. doi.org/10.1017/cbo9781107298019
Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media. doi.org/10.1007/978-0-387-84858-7
Bishop (2006). Pattern Recognition and Machine Learning. Springer Science+Business Media. doi.org/10.1007/978-0-387-45528-0

Cite this entry

@misc{dictml_trainerr,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {training error},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-13},
  url = {https://dictionaryofml.org/terms/trainerr.html}
}