Python demo — a script that recomputes what this entry states and prints one line per check
A validation set consists of data points which have
not been used for model training. For a learned
hypothesis, the average loss on the validation set,
the validation error, indicates how well the hypothesis
predicts the labels of data points outside the
training set. The validation error is used for model selection: the
hypotheses learned by different machine learning (ML) methods are
compared, and the one with the smallest validation error is selected.
The selected hypothesis depends on the validation set, so
assessing it requires a test set whose data points
entered neither training nor selection. The size of the
validation set determines how reliable the validation error is, and
Hoeffding's inequality prescribes the size needed for the
risk to lie, with high probability, within a given
uncertainty band around the measured validation error. When
data points are scarce, $k$-fold cross-validation ($k$-fold CV) averages the
validation errors obtained from using different folds of the
dataset as validation set.
Definition
B-valsetConsider three days of weather
recordings: for each day, the morning minimum temperature is the
feature and the maximum daytime temperature is the
label (Jung, 2022, Ch. 1).
Fig. 1 shows a training set
$\trainset$ of three such days along with two hypotheses
learned from it: the straight line $\learnthypothesis^{(1)}$
delivered by empirical risk minimization (ERM) on the linear model, and a degree-two
polynomial $\learnthypothesis^{(2)}$ that passes through all
three data points and therefore attains a training error of
zero. Three further days, marked by open triangles, were held
back from model training: they form a validation
set, on which the loss of each learned hypothesis
can be evaluated.
Figure 1: Data points representing daily weather conditions
in Finland: each data point is a day, with the morning
minimum temperature as its feature and the maximum
daytime temperature as its label. The three filled
circles form the training set $\trainset$; both curves are
hypotheses learned from $\trainset$. The three open
triangles were held back from model training and
form a validation set $\valset$: the average loss on
them estimates how well each hypothesis predicts the
labels of data points that were not used to
learn it.
Data generated by pythondemos/valset.py
A validation set $\valset$ consists of data points which
have not been used for model training. For a
hypothesis $\learnthypothesis$ learned from the
training set, the resulting average loss on the
validation set,
\[
\frac{1}{|\valset|} \sum_{\datapoint \in \valset}
\lossfunc{\datapoint}{\learnthypothesis} \text{,}
\]
is the validation error: it indicates how well $\learnthypothesis$
predicts the labels of data points outside the
training set (Jung, 2022, Sect. 6.2). In
Fig. 1, the polynomial
$\learnthypothesis^{(2)}$ attains a training error of zero but a
validation error of $17.3$, while the line $\learnthypothesis^{(1)}$
has a training error of $2.9$ and a validation error of $3.0$.
B-kfoldThe validation error is used for model selection: the
hypotheses learned by different machine learning (ML) methods are
compared, and the one with the smallest validation error is selected
(Jung, 2022, Sect. 6.3). Fig. 2
shows the selection between the two learned hypotheses of
the weather example: the line is selected, since $3.0 < 17.3$.
The selected hypothesis depends on $\valset$: its
validation error is the smallest of the compared validation errors, so
the guarantee for a single fixed hypothesis no longer
applies, and a valid bound must hold uniformly over all compared
candidates, growing with their number
(Shalev-Shwartz and Ben-David, 2014, Sect. 11.2). Assessing the selected
hypothesis therefore requires a third dataset whose
data points entered neither training nor selection:
the test set (Hastie et al., 2009, Sect. 7.2). In Fig. 2, the
three days of the test set, marked by open squares, yield a
test error of $1.3$ for the selected line.
Figure 2: Model selection with a validation set. The two curves are
the hypotheses of Fig. 1;
the validation error over the open triangles selects the line
($3.0$ against $17.3$ for the polynomial), drawn thick. The
selected hypothesis depends on $\valset$, so a third
set of days that entered neither training nor
selection, the test set (open squares), assesses it:
the selected line incurs a test error of $1.3$.
Data generated by pythondemos/valset.py
The size $|\valset|$ determines how well the validation error
reflects the overall performance of a learned hypothesis:
the validation error is an average of $|\valset|$ terms, and a
validation set that is too small results in an unreliable
validation error (Jung, 2022, Sect. 6.2.1). A
concentration inequality quantifies the reliability. For a
loss with values in $[0,1]$ and a fixed hypothesis
$\learnthypothesis$, Hoeffding's inequality
(Shalev-Shwartz and Ben-David, 2014, Thm. 11.1) guarantees that, with
probability at least $1 - \delta$, the risk of $\learnthypothesis$ lies
within an uncertainty band of half-width $\Delta$ around the
measured validation error, provided that
\[
|\valset| \geq \frac{\ln(2/\delta)}{2 \Delta^{2}} \text{.}
\]
Fig. 3 shows this lower bound as a
function of the half-width $\Delta$ for three values of $\delta$.
Narrowing the band is costly: halving $\Delta$ quadruples the
required size. To have the validation error within $\Delta = 0.1$ of
the risk with $\delta = 0.05$, it suffices to hold back
$|\valset| \geq 185$ data points, regardless of the
hypothesis space and of the size of $\trainset$.
Figure 3: The lower bound $\ln(2/\delta)/(2 \Delta^{2})$ on the
size $|\valset|$ as a function of the half-width $\Delta$ of
the uncertainty band around the measured validation error, for
three values of the confidence parameter $\delta$. The
vertical axis is logarithmic: halving $\Delta$ quadruples the
required size, while tightening $\delta$ from $0.5$ to $0.01$
raises it by a factor below four
When data points are scarce, holding back a large
validation set leaves few data points for the
training set. As a remedy, $k$-fold cross-validation ($k$-fold CV) divides the
dataset into $k$ folds and uses each fold in turn as
validation set: every fold yields a noisy validation error, an
average over few data points, and averaging the $k$
per-fold validation errors produces a more reliable estimate of the
expected loss
(Jung, 2022, Sect. 6.2.2; Stone, 1974).
In a complete workflow, the available dataset is thus
split three ways: a training set to learn hypotheses,
a validation set to select among them, and a test set, used
only once, to assess the selected hypothesis.
See also: validation, training set, test set,
data point, risk, hypothesis, loss,
validation error, training error, $k$-fold cross-validation, leave-one-out cross-validation,
model selection.
Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. Cambridge Univ. Press. doi.org/10.1017/cbo9781107298019
Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media. doi.org/10.1007/978-0-387-84858-7
Stone (1974). Cross-Validatory Choice and Assessment of Statistical Predictions. Journal of the Royal Statistical Society. Series B (Methodological).
Cite this entry
@misc{dictml_valset,
author = {Jung, Alexander and Olioumtsevits, Konstantina and Schnoor, Ekkehard},
title = {validation set},
howpublished = {Dictionary of Applied Machine Learning (course edition)},
year = {2026},
doi = {10.5281/zenodo.21569296},
note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-09-11},
url = {https://dictionaryofml.org/terms/valset.html}
}
