Dictionary of Applied Machine Learning: 107 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

validation set

Updated on 2026-10-22

Readiness 100/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Cite this entry

See also training set test set data point hypothesis loss validation error training error $k$-fold cross-validation model selection

A validation set consists of data points that have not been used for model training. For a learned hypothesis, the average loss on the validation set, the validation error, indicates how well the hypothesis predicts the labels of data points outside the training set. The validation error is used for model selection: the hypotheses learned by different machine learning (ML) methods are compared, and the one with the smallest validation error is selected. The selected hypothesis depends on the validation set, so assessing it requires a test set whose data points entered neither training nor selection. The size of the validation set determines how reliable the validation error is. When data points are scarce, $k$-fold cross-validation ($k$-fold CV) uses each fold of the dataset in turn as validation set and averages the $\nrfolds$ validation errors.

Figure produced by the Python demo of the entry validation set
Figure written by the entry's Python demo valset.py, which recomputes what the entry states: the script block by block · open in Colab

Definition

B-dataConsider three days of weather recordings: for each day, the morning minimum temperature is the feature and the maximum daytime temperature is the label. Fig. 1 shows a training set $\trainset$ of three such days along with two hypotheses learned from it: the linear hypothesis $\learnthypothesis^{(1)}$ delivered by empirical risk minimization (ERM) on the linear model, and a degree-two polynomial $\learnthypothesis^{(2)}$ that passes through all three data points and therefore attains a training error of zero.

B-valsetThree further days, marked by open triangles, were held back from model training: they form a validation set $\valset$. The validation error of a hypothesis $\learnthypothesis$ learned from the training set is its average loss on the validation set (Shalev-Shwartz and Ben-David, 2014, Sect. 11.2.1), \[ \frac{1}{|\valset|} \sum_{\datapoint \in \valset} \lossfunc{\datapoint}{\learnthypothesis} \text{.} \] Here, $\lossfunc{\datapoint}{\learnthypothesis}$ is the loss of $\learnthypothesis$ on the data point $\datapoint$. In Fig. 1, the polynomial $\learnthypothesis^{(2)}$ attains a training error of zero but a validation error of $17.3$, while the linear hypothesis $\learnthypothesis^{(1)}$ has a training error of $2.9$ and a validation error of $3.0$.

Figure 1 of the entry valset
Figure 1: Data points representing daily weather conditions. Each data point is a day, with the morning minimum temperature as its feature $\feature$ and the maximum daytime temperature as its label $\truelabel$. The three filled circles form the training set $\trainset$; the solid curve is the linear hypothesis $\learnthypothesis^{(1)}$ and the dashed polynomial the hypothesis $\learnthypothesis^{(2)}$, both learned from $\trainset$. The three open triangles were held back from model training and form a validation set $\valset$: the average loss on them, $3.0$ for $\learnthypothesis^{(1)}$ and $17.3$ for $\learnthypothesis^{(2)}$, is the validation error of each hypothesis. Data generated by pythondemos/valset.py
The comparison of the two validation errors is an instance of model selection: the hypotheses learned by different machine learning (ML) methods are compared, and the one with the smallest validation error is selected (Bishop, 2006, Sect. 1.3). In Fig. 1, the hypothesis $\learnthypothesis^{(1)}$ is selected, since $3.0 < 17.3$. The selected hypothesis depends on $\valset$: its validation error is the smallest of the compared validation errors, so the validation error of the selected hypothesis is biased downward. A bound that covers the selected hypothesis must hold for all compared hypotheses at once, which requires a validation set larger than the size that Hoeffding's inequality suggests for one hypothesis (Shalev-Shwartz and Ben-David, 2014, Thm. 11.2).

B-modelselAssessing the hypothesis learned from the model that was chosen on the basis of the validation errors requires a third dataset whose data points entered neither training nor selection: the test set $\testset$ (Hastie et al., 2009, Sect. 7.2). In Fig. 2, the three days of the test set, marked by open squares, yield a test error $\testerror = 1.3$ for the selected hypothesis $\learnthypothesis^{(1)}$. The test error is the only one of the three numbers that was not used to choose anything, so it is not biased by selection. With only three test days, however, it is itself a noisy estimate of the risk of $\learnthypothesis^{(1)}$.

Figure 2 of the entry valset
Figure 2: Model selection with a validation set. The thick solid curve is the selected hypothesis $\learnthypothesis^{(1)}$ and the dashed polynomial the rejected hypothesis $\learnthypothesis^{(2)}$, both from Fig. 1; the validation error over the validation set $\valset$ (open triangles) selects the hypothesis $\learnthypothesis^{(1)}$ ($3.0$ against $17.3$ for the polynomial $\learnthypothesis^{(2)}$). The selected hypothesis depends on $\valset$, so a third set of days that entered neither training nor selection, the test set $\testset$ (open squares), assesses it: the selected hypothesis $\learnthypothesis^{(1)}$ incurs a test error $\testerror = 1.3$. Data generated by pythondemos/valset.py
The size $|\valset|$ determines how well the validation error reflects the overall performance of a learned hypothesis: the validation error is an average of $|\valset|$ losses, one per validation data point, and a validation set that is too small results in an unreliable validation error (Bishop, 2006, Sect. 1.3).

B-verifyFor a loss with values in $[0,1]$ and a fixed hypothesis $\learnthypothesis$, Hoeffding's inequality (Shalev-Shwartz and Ben-David, 2014, Thm. 11.1) guarantees that, with probability at least $1 - \delta$ for a confidence parameter $\delta \in (0, 1)$ chosen by the user, the risk of $\learnthypothesis$ lies within an uncertainty band of half-width $\epsilon$ around the measured validation error. The guarantee requires that \[ |\valset| \geq \frac{\ln(2/\delta)}{2 \epsilon^{2}} \text{.} \] Fig. 3 shows this lower bound as a function of the half-width $\epsilon$ for three values of $\delta$; it holds for any validation error, not only those of the weather example. Narrowing the band is costly: halving $\epsilon$ quadruples the required size. To have the validation error within $\epsilon = 0.1$ of the risk with $\delta = 0.1$, it suffices to hold back $|\valset| \geq 150$ data points, the point of the dashed curve at $\epsilon = 0.1$ in Fig. 3, regardless of the hypothesis space and of the size of $\trainset$.

Figure 3 of the entry valset
Figure 3: The lower bound $\ln(2/\delta)/(2 \epsilon^{2})$ on the size $|\valset|$ as a function of the half-width $\epsilon$ of the uncertainty band around the measured validation error, for three values of the confidence parameter $\delta$: $0.01$ (solid curve), $0.1$ (dashed curve), and $0.5$ (dotted curve). The vertical axis is logarithmic. Halving $\epsilon$ from $0.1$ to $0.05$ quadruples the required size (filled circles on the dashed curve, $150$ to $599$), while tightening $\delta$ from $0.5$ to $0.01$ at $\epsilon = 0.1$ raises it by a factor around four (open circles, $69$ to $265$)
The bound can be checked by simulation. Fig. 4 reports how often the validation error of a fixed hypothesis deviates from the risk by more than $\epsilon$, when the validation set holds exactly the number of data points that the bound requires. The loss used there takes the values $0$ and $1$ with equal probability, so its variance is the largest a loss with values in $[0,1]$ can have. Each share is measured over $200{,}000$ validation sets. Every measured share stays below its $\delta$, and below $\delta / 4$, so the size the bound requires is larger than the size the measured deviations call for. A validation set eight times smaller does fail the guarantee: at $\epsilon = 0.1$ and $\delta = 0.1$, where the bound asks for $150$ data points, $19$ of them leave $36$ of every $100$ validation errors farther than $\epsilon$ from the risk.
Figure 4 of the entry valset
Figure 4: Measured share of validation sets whose validation error deviates from the risk by more than the half-width $\epsilon$, when the validation set holds exactly the $\lceil \ln(2/\delta) / (2 \epsilon^{2}) \rceil$ data points that Fig. 3 requires. Each share is measured over $200{,}000$ validation sets, for a loss with values in $\{0, 1\}$ and risk $1/2$. The three confidence parameters of Fig. 3 keep their styles: $\delta = 0.01$ (solid curve), $0.1$ (dashed curve), and $0.5$ (dotted curve). The three gray segments at the right margin, labeled with their $\delta$, mark the nominal values. Every measured share stays below its $\delta$ by a factor of at least four. The vertical axis is logarithmic. Data generated by pythondemos/valset.py
When data points are scarce, holding back a large validation set leaves few data points for the training set. As a remedy, $k$-fold cross-validation ($k$-fold CV) divides the dataset into $\nrfolds$ folds and, for each fold in turn, uses that fold as validation set and the other $\nrfolds - 1$ folds as training set, so each of the $\nrfolds$ validation errors is computed on different data points and for a different learned hypothesis. Every fold yields a noisy validation error, an average of the losses over the few data points of the fold, and averaging the $\nrfolds$ per-fold validation errors produces a more reliable estimate of the risk (Hastie et al., 2009, Sect. 7.10.1; Stone, 1974).

Taken together, a typical ML workflow splits the available dataset three ways: a training set to learn hypotheses, a validation set to select among them (or the folds of $k$-fold CV in its place), and a test set, used only once, to assess the selected hypothesis.

See also: validation, training set, test set, data point, risk, hypothesis, loss, validation error, training error, $k$-fold cross-validation, leave-one-out cross-validation, model selection.

References

  1. Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. Cambridge Univ. Press. doi.org/10.1017/cbo9781107298019
  2. Bishop (2006). Pattern Recognition and Machine Learning. Springer Science+Business Media. doi.org/10.1007/978-0-387-45528-0
  3. Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media. doi.org/10.1007/978-0-387-84858-7
  4. Stone (1974). Cross-Validatory Choice and Assessment of Statistical Predictions. Journal of the Royal Statistical Society. Series B (Methodological). doi.org/10.1111/j.2517-6161.1974.tb00994.x

Cite this entry

@misc{dictml_valset,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {validation set},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-10-27},
  url = {https://dictionaryofml.org/terms/valset.html}
}