Dictionary of Applied Machine Learning

training set

Updated on 2026-09-04

Typeset PDF version — the authoritative form of this entry

Python demo — a script that recomputes what this entry states and prints one line per check

The ultimate goal of machine learning (ML) is to learn a hypothesis that accurately predicts the label of any data point based on its features. A training set is a dataset used to compare the performance of candidate hypotheses: it consists of data points for which the loss incurred by each candidate can be evaluated. Model training methods then pick the hypothesis that performs best on the training set. For example, empirical risk minimization (ERM) learns the hypothesis minimizing the average loss on the training set, the empirical risk; the minimum value itself is the training error. Since the learned hypothesis is picked to perform well on the training set, a small training error can be misleading and result in overfitting. The validation error on a held-back validation set probes whether the learned hypothesis also predicts well outside the training set.

Definition

The ultimate goal of a machine learning (ML) method is to learn a hypothesis (or to train a model) that incurs a small prediction error for any data point. The prediction error is measured by some loss function, and the goal is formalized probabilistically as a small risk: the expected loss under the probability distribution from which the data points are drawn (independent and identically distributed assumption (i.i.d. assumption)). The method must therefore compare the candidate hypotheses from its hypothesis space on data points for which the loss can be evaluated. For example, in regression or classification, the loss can only be computed for data points with a known label. These data points form a dataset referred to as the training set, $\trainset = \{\datapoint^{(\sampleidx)}\}_{\sampleidx=1}^{\samplesize}$.

For example, empirical risk minimization (ERM) minimizes the average loss on the training set, the empirical risk, to learn a hypothesis \[ \learnthypothesis \in \argmin_{\hypothesis \in \hypospace} \frac{1}{\samplesize} \sum_{\sampleidx=1}^{\samplesize} \lossfunc{\datapoint^{(\sampleidx)}}{\hypothesis} \text{.} \] The empirical risk of the learned hypothesis is the training error. More generally, the training set is the input of the map $\algomap$ that defines a training method: $\learnthypothesis = \algomap(\trainset)$, and training is to compute this map.

B-polyConsider three days of weather recordings: for each day, the morning minimum temperature is the feature and the maximum daytime temperature is the label (Jung, 2022, Ch. 1). Fig. 1 shows a training set of these three days along with two hypotheses learned from it: the straight line $\learnthypothesis^{(1)}$ delivered by ERM on the linear model, and a degree-two polynomial $\learnthypothesis^{(2)}$ that passes through all three data points and therefore attains a training error of zero.

Figure 1 of the entry trainset
Figure 1: Data points representing daily weather conditions in Finland: each data point is a day, with the morning minimum temperature as its feature and the maximum daytime temperature as its label. The three filled circles form the training set $\trainset$. Both curves are hypotheses learned from $\trainset$: the solid straight line $\learnthypothesis^{(1)}$ minimizes the average squared error on $\trainset$ over the linear model, while the dashed degree-two polynomial $\learnthypothesis^{(2)}$ passes through all three data points and attains a training error of zero. Both incur a small average loss on $\trainset$, but the two curves differ sharply outside the training set. Data generated by pythondemos/trainset.py
Which data points are contained in $\trainset$ decides what is learned. The choice of the training set is therefore crucial for trustworthy artificial intelligence (trustworthy AI): a training set that underrepresents a group of data points yields a learned hypothesis that incurs a larger loss on that group, a source of unfairness (see fairness) (Buolamwini and Gebru, 2018; Barocas et al., 2019; Mehrabi et al., 2021). Since $\learnthypothesis$ is chosen to make this particular average small, the training error understates the loss that the same hypothesis incurs on data points outside $\trainset$ (Jung, 2022, Sect. 6.2). The training error is therefore a poor estimate of the loss incurred outside the training set. Fig. 1 shows how misleading it can be: the polynomial $\learnthypothesis^{(2)}$ has a smaller training error than the straight line, yet the two curves differ sharply away from the training data points, which is where the loss on new data points is incurred (see overfitting).

B-misleadingA second subset of data points, the validation set $\valset$, is therefore held back from model training (Fig. 2): the average loss on $\valset$, the validation error, probes the learned hypothesis on data points it was not trained on. Comparing the two errors diagnoses the ML method: if both are small, the learned hypothesis also predicts well outside $\trainset$; a small training error paired with a validation error far above it means the hypothesis space is too large compared to the number of data points in $\trainset$ (Jung, 2022, Sect. 6.6; Hastie et al., 2009).

Figure 2 of the entry trainset
Figure 2: The available dataset split into two disjoint subsets. Only the shaded subset, the training set $\trainset$, enters the training method $\algomap$ that delivers a learned hypothesis $\learnthypothesis$; the validation set is what remains to estimate how that hypothesis behaves on data points it was not trained on
See also: training, dataset, validation set, test set, data point, ERM, hypothesis, loss, training error, validation error, hypothesis space, overfitting.

References

  1. Jung (2022). Machine Learning: The Basics. Springer Nature. doi.org/10.1007/978-981-16-8193-6
  2. Buolamwini and Gebru (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. Conference on Fairness, Accountability and Transparency.
  3. Barocas et al. (2019). Fairness and Machine Learning. fairmlbook.org. fairmlbook.org/
  4. Mehrabi et al. (2021). A Survey on Bias and Fairness in Machine Learning. ACM Computing Surveys. doi.org/10.1145/3457607
  5. Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media. doi.org/10.1007/978-0-387-84858-7

Cite this entry

@misc{dictml_trainset,
  author = {Jung, Alexander and Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {training set},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-09-11},
  url = {https://dictionaryofml.org/terms/trainset.html}
}