# training set

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-02. [HTML](https://dictionaryofml.org/terms/trainset.html) · [typeset PDF](https://dictionaryofml.org/terms/trainset.pdf) · [LaTeX](https://dictionaryofml.org/terms/trainset.tex) · [Python demo](https://dictionaryofml.org/terms/trainset.py) · [notebook](https://dictionaryofml.org/terms/trainset.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A training set is a dataset used to train a model: it consists of data points for which the loss incurred by each candidate hypothesis can be evaluated, so that training methods can pick the hypothesis that performs best on it. Empirical risk minimization, for example, learns the hypothesis minimizing the average loss on the training set, the empirical risk, whose minimum value is the training error. Because the learned hypothesis is picked to perform well on the training set, a small training error can be misleading and result in overfitting. The validation error on a held-back validation set indicates whether the learned hypothesis also predicts well outside the training set.

## Definition

Consider three days of weather recordings: for each day, the morning minimum temperature is the feature and the maximum daytime temperature is the label. The ultimate goal of a machine learning method is to learn a hypothesis (or to train a model) that incurs a small prediction error not only on the data points used for training but on any data point that is typical for the application. The prediction error is measured by some loss function: the loss is a number, typically nonnegative, that quantifies the error of a hypothesis on a single data point, such as the squared difference between the predicted and the true label. The goal is formalized probabilistically as a small risk: the expected loss under the probability distribution from which the data points are drawn (independent and identically distributed assumption). The method must therefore compare the candidate hypotheses from its hypothesis space on data points for which the loss can be evaluated. For example, in regression or classification, the loss can only be computed for data points with a known label. These data points form a dataset referred to as the training set, $\mathcal{D}^{(\rm train)} = \{{\mathbf z}^{(r)}\}_{r=1}^{m}$.

A training method is a map $\mathcal{A}$ whose input is the training set and whose output is a learned hypothesis, $\widehat{h} = \mathcal{A}(\mathcal{D}^{(\rm train)})$. Training evaluates this map on the given training set. For example, empirical risk minimization minimizes the average loss on the training set, the empirical risk, to learn a hypothesis

$$
\widehat{h} \in \mathop{\mathrm{arg\,min}}_{h \in \mathcal{H}}
\frac{1}{m}
\sum_{r=1}^{m}
L\left({\mathbf z}^{(r)},h \right) \text{.}
$$

The empirical risk of the learned hypothesis is the training error.

Fig. 1 shows a training set of the three days along with two hypotheses learned from it: the linear hypothesis $\widehat{h}^{(1)}$ delivered by empirical risk minimization on the linear model, and a degree-two polynomial $\widehat{h}^{(2)}$ that passes through all three data points and therefore attains a training error of zero.

![Figure 1 of the entry trainset](https://dictionaryofml.org/terms/fig/trainset-fig1.svg)

*Figure 1: Data points representing daily weather conditions: each data point is a day, with the morning minimum temperature as its feature and the maximum daytime temperature as its label. The three filled circles form the training set $\mathcal{D}^{(\rm train)}$. Both curves are hypotheses learned from $\mathcal{D}^{(\rm train)}$: the linear hypothesis $\widehat{h}^{(1)}$ (solid curve) minimizes the average squared error on $\mathcal{D}^{(\rm train)}$ over the linear model, while the dashed degree-two polynomial $\widehat{h}^{(2)}$ passes through all three data points and attains a training error of zero. Both incur a small average loss on $\mathcal{D}^{(\rm train)}$, but between and beyond the three circles the two curves are far apart: at the largest morning minimum temperature shown, the two predicted maximum temperatures differ by about seven degrees. Data generated by pythondemos/trainset.py*

Since $\widehat{h}$ is chosen to make the empirical risk small, the training error is typically smaller than the loss that the same hypothesis incurs on data points outside $\mathcal{D}^{(\rm train)}$ (Hastie et al., 2009, Sect. 7.4). A small training error therefore does not guarantee a small loss outside the training set. Fig. 1 illustrates this: the polynomial $\widehat{h}^{(2)}$ attains a training error of zero, the linear hypothesis $\widehat{h}^{(1)}$ does not. The figure shows no data points outside $\mathcal{D}^{(\rm train)}$, but it shows that the two curves predict different labels for feature values between and beyond the three training data points. At most one of the two curves can therefore incur a small loss on a new data point with such a feature value, and the training error cannot tell which one (see overfitting).

A second subset of data points, the validation set $\mathcal{D}^{(\rm val)}$, is therefore held back from model training (Fig. 2): the average loss on $\mathcal{D}^{(\rm val)}$, the validation error, evaluates the learned hypothesis on data points it was not trained on. Comparing the two errors diagnoses the machine learning method: if both are small, the learned hypothesis is likely to predict well outside $\mathcal{D}^{(\rm train)}$ as well. A small training error paired with a validation error far above it indicates overfitting; one typical cause is a hypothesis space that is too large for the number of data points in $\mathcal{D}^{(\rm train)}$ (Bishop, 2006, Sect. 1.1; Hastie et al., 2009, Sect. 7.2).

![Figure 2 of the entry trainset](https://dictionaryofml.org/terms/fig/trainset-fig2.svg)

*Figure 2: The available dataset split into two disjoint subsets. Only the shaded subset, the training set $\mathcal{D}^{(\rm train)}$, enters the training method $\mathcal{A}$ that delivers a learned hypothesis $\widehat{h}$; the validation set is what remains to estimate how that hypothesis behaves on data points it was not trained on*

Which data points are contained in $\mathcal{D}^{(\rm train)}$ decides what is learned. The choice of the training set therefore matters for trustworthy artificial intelligence: a training set that underrepresents a group of data points can yield a learned hypothesis that incurs a larger loss on that group, a source of unfairness (see fairness) (Barocas et al., 2023; Buolamwini and Gebru, 2018; Mehrabi et al., 2021).

**See also:** [training](https://dictionaryofml.org/terms/training.html), [dataset](https://dictionaryofml.org/terms/dataset.html), [validation set](https://dictionaryofml.org/terms/valset.html), [test set](https://dictionaryofml.org/terms/testset.html), [data point](https://dictionaryofml.org/terms/datapoint.html), [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), [hypothesis](https://dictionaryofml.org/terms/hypothesis.html), [loss](https://dictionaryofml.org/terms/loss.html), [training error](https://dictionaryofml.org/terms/trainerr.html), [validation error](https://dictionaryofml.org/terms/valerr.html), [hypothesis space](https://dictionaryofml.org/terms/hypospace.html), [overfitting](https://dictionaryofml.org/terms/overfitting.html).

## References

- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Barocas et al. (2023). Fairness and Machine Learning: Limitations and Opportunities. *MIT Press*. <https://fairmlbook.org/>
- Buolamwini and Gebru (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. *Proceedings of the 1st Conference on Fairness, Accountability and Transparency*. <https://proceedings.mlr.press/v81/buolamwini18a.html>
- Mehrabi et al. (2021). A Survey on Bias and Fairness in Machine Learning. *ACM Computing Surveys*. <https://doi.org/10.1145/3457607>

## Cite this entry

```bibtex
@misc{dictml_trainset,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {training set},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/trainset.html}
}
```