# test set

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/testset.html) · [typeset PDF](https://dictionaryofml.org/terms/testset.pdf) · [LaTeX](https://dictionaryofml.org/terms/testset.tex) · [Python demo](https://dictionaryofml.org/terms/testset.py) · [notebook](https://dictionaryofml.org/terms/testset.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A test set contains data points used to evaluate a hypothesis delivered by a machine learning method. These points are used neither for training nor for model selection. A test set is therefore also called a hold-out set. Under the independent and identically distributed assumption, its average loss is an unbiased estimator of the risk of the learned hypothesis. A concentration inequality relates the accuracy of this estimate to the test set size. The estimate remains valid only if no design choice depends on the test set. Selection based on repeated evaluation converts it into a validation set; overlap with the training set is test set contamination.

## Definition

Consider a machine learning method that uses weather recordings from Krems an der Donau (Austria). It learns a hypothesis $\widehat{h}$ that predicts tomorrow's maximum temperature from today's maximum. A training set is used to learn a hypothesis via empirical risk minimization. A validation set is then used to compare hypotheses from different models, such as a linear model and polynomial regression. Both sets influence the selected hypothesis, so its loss on them is usually smaller than on data points that the method did not use (Hastie et al., 2009, Sect. 7.4; Shalev-Shwartz and Ben-David, 2014, Sect. 11.2.2). Assessing $\widehat{h}$ therefore requires data points from neither set.

A test set $\mathcal{D}^{(\rm test)}$ is a dataset of data points used to evaluate a hypothesis delivered by a machine learning method. Its data points enter neither the training of a model on the training set $\mathcal{D}^{(\rm train)}$ nor model selection based on a validation set $\mathcal{D}^{(\rm val)}$ (Fig. 1). A test set is therefore also called a hold-out set. Some texts instead use that name for the validation set (Shalev-Shwartz and Ben-David, 2014, Sect. 11.2). For the temperature forecaster, the test set consists of days absent from both other sets.

![Figure 1 of the entry testset](https://dictionaryofml.org/terms/fig/testset-fig1.svg)

*Figure 1: Three roles for data points. On the training set (filled circles), two hypotheses are learned via empirical risk minimization from different models: $\widehat{h}^{(1)}$ from a linear model (solid curve) and $\widehat{h}^{(2)}$ from degree-three polynomial regression (dashed curve). The validation set (open triangles) is used to select among them. The test set (open squares) enters neither training nor model selection*

Suppose that $\widehat{h}$ was learned and selected without using $\mathcal{D}^{(\rm test)}$. Its test error is the average loss

$$
\frac{1}{|\mathcal{D}^{(\rm test)}|} \sum_{{\mathbf z} \in \mathcal{D}^{(\rm test)}}
L\left({\mathbf z},\widehat{h} \right) \text{.}
$$

Here, $L\left({\mathbf z},\widehat{h} \right)$ is the loss, a number that quantifies the error on one data point. Under the independent and identically distributed assumption, the points in $\mathcal{D}^{(\rm train)}$, $\mathcal{D}^{(\rm val)}$, and $\mathcal{D}^{(\rm test)}$ are independent draws from the same probability distribution $p({\mathbf z})$. The test error is then an unbiased estimator of the risk of $\widehat{h}$ (Hastie et al., 2009). The risk is the expected loss on a data point drawn from $p({\mathbf z})$ independently of all three sets. The validation error is used to select $\widehat{h}$, so it tends to understate the risk of the selected hypothesis.

The sizes of the three sets are design choices. Under the independent and identically distributed assumption, the test error averages $|\mathcal{D}^{(\rm test)}|$ independent loss values. If the loss lies in $[0,1]$, a concentration inequality bounds the difference between test error and risk. For a confidence level $\delta \in (0,1)$, this difference is at most $\sqrt{\ln(2/\delta)/(2\,|\mathcal{D}^{(\rm test)}|)}$ with probability at least $1-\delta$ (Shalev-Shwartz and Ben-David, 2014, Thm. 11.1). Solving that bound for $|\mathcal{D}^{(\rm test)}|$ gives the test set size a desired difference and confidence level require.

The validation set needs a concentration inequality that covers every hypothesis compared during model selection at once, so its required size grows with their number (Shalev-Shwartz and Ben-David, 2014, Thm. 11.2). The training set receives the remaining data points. No fixed rule determines the proportions. Fig. 2 shows a common example: 50 percent for training and 25 percent each for validation and testing (Hastie et al., 2009, Sect. 7.2).

![Figure 2 of the entry testset](https://dictionaryofml.org/terms/fig/testset-fig2.svg)

*Figure 2: A dataset $\mathcal{D}$ split into the training set $\mathcal{D}^{(\rm train)}$, the validation set $\mathcal{D}^{(\rm val)}$, and the test set $\mathcal{D}^{(\rm test)}$, with the typical proportions of Hastie et al. (2009, Sect. 7.2)*

A test error remains a valid risk estimate only if the test set is used once, after all choices are fixed. Using $\mathcal{D}^{(\rm test)}$ to compare models converts it into a validation set, and its error into a validation error. Moving any of its data points into the training set is test set contamination (Goodfellow et al., 2016, Sect. 5.3).

In a machine learning competition, organizers compare models submitted by participants on a common test set (Hastie et al., 2009, Sect. 11.9). The organizers keep the test labels private and compute each error. Participants cannot use those labels to revise their models. The reported test error can therefore estimate risk, provided that the organizers limit repeated feedback on the same test set.

**Synonyms:** hold-out set.

**See also:** [training set](https://dictionaryofml.org/terms/trainset.html), [validation set](https://dictionaryofml.org/terms/valset.html), [model selection](https://dictionaryofml.org/terms/modelsel.html), risk, [generalization](https://dictionaryofml.org/terms/generalization.html), test set contamination, [data leakage](https://dictionaryofml.org/terms/dataleakage.html), [empirical risk minimization](https://dictionaryofml.org/terms/erm.html).

## References

- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Goodfellow et al. (2016). Deep Learning. *MIT Press*.

## Cite this entry

```bibtex
@misc{dictml_testset,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {test set},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/testset.html}
}
```