Dictionary of Applied Machine Learning

test set

Updated on 2026-09-04

Typeset PDF version — the authoritative form of this entry

A test set is a dataset of data points used neither for training a model nor for choosing between candidate models based on a validation set. For a hypothesis learned and selected without reference to the test set, the average loss on it estimates the risk under the independent and identically distributed assumption (i.i.d. assumption). This distinguishes it from the validation set, whose average loss flows back into the choice of the hypothesis and is therefore optimistic for the chosen candidate. The estimate stays valid only as long as no design choice depends on the test set; repeated evaluation with selection turns it into a validation set, and overlap with the training set constitutes test set contamination.

Definition

A straight line and a degree-two polynomial have both been fitted to historic days of weather recordings, and the line was selected because it incurred the smaller average loss on days held back from the fit (model selection via a validation set). Quoting that same average as the line's expected error on future days would understate the error: the line was selected for scoring well on exactly those held-back days. The days reserved to answer the final question — how large a loss to expect on a new day — form the test set.

A test set $\testset$ is a dataset of data points used neither for training a model, e.g., via empirical risk minimization (ERM) on the training set $\trainset$, nor for choosing between candidate models based on a validation set $\valset$ (Fig. 1). For a hypothesis $\learnthypothesis$ that was learned and selected without reference to $\testset$, the test error is the average loss \[ \frac{1}{|\testset|} \sum_{\datapoint \in \testset} \lossfunc{\datapoint}{\learnthypothesis} \text{.} \] It estimates the risk of $\learnthypothesis$ under the independent and identically distributed assumption (i.i.d. assumption) (Hastie et al., 2009). The distinction from the validation set is the direction of information flow: the validation error flows back into the choice of $\learnthypothesis$, so it is an optimistic estimate for the chosen candidate; the test error is computed once, after all choices are frozen, and flows back into nothing.

Figure 1 of the entry testset
Figure 1: Three roles for data points. The filled circles form the training set on which $\learnthypothesis$ is learned, the open triangles the validation set used to select among candidate models, and the open squares the test set: it enters neither training nor model selection, so the average loss on it estimates the risk of $\learnthypothesis$
The estimate stays valid only as long as the test set is used this way. Evaluating many candidate models on $\testset$ and keeping the best turns the test set into a validation set, and its error into a validation error; data points of $\testset$ that enter the training set constitute test set contamination. In an ML competition, the labels of the test set are therefore withheld from the participants: no design choice of theirs can then depend on the test data, and the reported error keeps its meaning as a risk estimate.

See also: training set, validation set, model selection, risk, generalization, test set contamination, data leakage, empirical risk minimization.

References

  1. Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media. doi.org/10.1007/978-0-387-84858-7

Cite this entry

@misc{dictml_testset,
  author = {Jung, Alexander and Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {test set},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-09-11},
  url = {https://dictionaryofml.org/terms/testset.html}
}