trainset.py, which recomputes what the entry states: the script block by block · open in ColabDictionary of Applied Machine Learning: 107 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here
Updated on 2026-10-22
Readiness 100/100
See also training dataset validation set test set data point empirical risk minimization hypothesis loss training error validation error hypothesis space overfitting
A training set is a dataset used to train a model: it consists of data points for which the loss incurred by each candidate hypothesis can be evaluated, so that training methods can pick the hypothesis that performs best on it. Empirical risk minimization (ERM), for example, learns the hypothesis minimizing the average loss on the training set, the empirical risk, whose minimum value is the training error. Because the learned hypothesis is picked to perform well on the training set, a small training error can be misleading and result in overfitting. The validation error on a held-back validation set probes whether the learned hypothesis also predicts well outside the training set.
trainset.py, which recomputes what the entry states: the script block by block · open in ColabThe ultimate goal of a machine learning (ML) method is to learn a hypothesis (or to train a model) that incurs a small prediction error not only on the data points used for training but on any data point that is typical for the application. The prediction error is measured by some loss function: the loss is a number, typically nonnegative, that quantifies the error of a hypothesis on a single data point, such as the squared difference between the predicted and the true label. The goal is formalized probabilistically as a small risk: the expected loss under the probability distribution from which the data points are drawn (independent and identically distributed assumption (i.i.d. assumption)). The method must therefore compare the candidate hypotheses from its hypothesis space on data points for which the loss can be evaluated. For example, in regression or classification, the loss can only be computed for data points with a known label. These data points form a dataset referred to as the training set, $\trainset = \{\datapoint^{(\sampleidx)}\}_{\sampleidx=1}^{\samplesize}$.
A training method is a map $\algomap$ whose input is the training set and whose output is a learned hypothesis, $\learnthypothesis = \algomap(\trainset)$. Training evaluates this map on the given training set. For example, empirical risk minimization (ERM) minimizes the average loss on the training set, the empirical risk, to learn a hypothesis \[ \learnthypothesis \in \argmin_{\hypothesis \in \hypospace} \frac{1}{\samplesize} \sum_{\sampleidx=1}^{\samplesize} \lossfunc{\datapoint^{(\sampleidx)}}{\hypothesis} \text{.} \] The empirical risk of the learned hypothesis is the training error.
B-ermConsider three days of weather recordings: for each day, the morning
minimum temperature is the feature and the maximum daytime
temperature is the label.
Fig. 1 shows a training set of these
three days along with two hypotheses learned from it: the
linear hypothesis $\learnthypothesis^{(1)}$ delivered by ERM on
the linear model, and a degree-two polynomial
$\learnthypothesis^{(2)}$ that passes through all three
data points and therefore attains a training error of zero.
pythondemos/trainset.py
B-polySince $\learnthypothesis$ is chosen to make the empirical risk small, the training error is typically smaller than the loss that the same hypothesis incurs on data points outside $\trainset$ (Hastie et al., 2009, Sect. 7.4). A small training error therefore does not guarantee a small loss outside the training set. Fig. 1 illustrates this: the polynomial $\learnthypothesis^{(2)}$ has a smaller training error than the linear hypothesis $\learnthypothesis^{(1)}$, so the training error prefers the polynomial. The figure shows no data points outside $\trainset$, but it shows that the two curves predict different labels for feature values between and beyond the three training data points. At most one of the two curves can therefore incur a small loss on a new data point with such a feature value, and the training error cannot tell which one (see overfitting).
B-misleadingA second subset of data points, the validation set $\valset$, is
therefore held back from model training
(Fig. 2): the average loss on $\valset$,
the validation error, probes the learned hypothesis on
data points it was not trained on. Comparing the two errors
diagnoses the ML method: if both are small, the learned
hypothesis is likely to predict well outside $\trainset$
as well. A small training error paired with a validation error far
above it indicates overfitting; one typical cause is a
hypothesis space that is too large for the number of
data points in $\trainset$
(Bishop, 2006, Sect. 1.1; Hastie et al., 2009, Sect. 7.2).
See also: training, dataset, validation set, test set, data point, empirical risk minimization, hypothesis, loss, training error, validation error, hypothesis space, overfitting.
@misc{dictml_trainset,
author = {Jung, Alexander},
editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
title = {training set},
howpublished = {Dictionary of Applied Machine Learning (course edition)},
year = {2026},
doi = {10.5281/zenodo.21569296},
note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-10-27},
url = {https://dictionaryofml.org/terms/trainset.html}
}