Dictionary of Applied Machine Learning: 107 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

test set

Updated on 2026-10-22

Readiness 100/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Cite this entry

See also training set validation set model selection generalization data leakage empirical risk minimization

A test set consists of data points used to evaluate the performance of a hypothesis delivered by a machine learning (ML) method. These data points are used neither for training nor for model selection, which is why a test set is also called a hold-out set. Under the independent and identically distributed assumption (i.i.d. assumption), the average loss on the test set is an unbiased estimator of the risk of the learned hypothesis, and a concentration inequality bounds its deviation from the risk in terms of the test-set size. This yields a guideline for splitting a dataset: make the test set and the validation set just large enough for the required precision and train on the rest. The estimate stays valid only as long as no design choice depends on the test set; repeated evaluation with selection turns it into a validation set, and overlap with the training set constitutes test set contamination.

Figure produced by the Python demo of the entry test set
Figure written by the entry's Python demo testset.py, which recomputes what the entry states: the script block by block · open in Colab

Definition

P-tuningConsider a machine learning (ML) method that reads in weather recordings from the station Krems an der Donau (Austria) and delivers a hypothesis $\learnthypothesis$ that predicts the next day's maximum temperature from the current day's. Assessing the quality of $\learnthypothesis$ requires applying it to data points that the method has not used: the method typically uses a training set for learning a hypothesis via empirical risk minimization (ERM) and a validation set for comparing the hypotheses learned from different models. The data points of these two sets have tuned the learned hypothesis, which therefore typically incurs a smaller loss on them than on other data points.

A test set $\testset$ is a dataset of data points used to evaluate the performance of a hypothesis delivered by an ML method. Its data points enter neither the training of a model, e.g., via ERM on the training set $\trainset$, nor the model selection based on a validation set $\valset$ (Fig. 1). Since its data points are held back from training and model selection, a test set is also referred to as a hold-out set; some texts use that name for the validation set instead (Shalev-Shwartz and Ben-David, 2014, Sect. 11.2).

P-sizeFor a hypothesis $\learnthypothesis$ that was learned and selected without reference to $\testset$, the test error is the average loss \[ \frac{1}{|\testset|} \sum_{\datapoint \in \testset} \lossfunc{\datapoint}{\learnthypothesis} \text{.} \] Interpreting the data points of $\trainset$, $\valset$, and $\testset$ as realizations of independent and identically distributed (i.i.d.) random variables (RVs) with a common probability distribution $p(\datapoint)$ (the independent and identically distributed assumption (i.i.d. assumption)), the test error is an unbiased estimator of the risk of $\learnthypothesis$ (Hastie et al., 2009). The distinction from the validation set is the direction of information flow: the validation error flows back into the choice of $\learnthypothesis$, so it is an optimistic estimate for the chosen candidate. The test error is computed once, after all choices are frozen, and flows back into nothing.

Figure 1 of the entry testset
Figure 1: Three roles for data points. On the training set (filled circles), two hypotheses are learned via ERM from different models: $\learnthypothesis^{(1)}$ from a linear model (solid) and $\learnthypothesis^{(2)}$ from degree-three polynomial regression (dashed). The validation set (open triangles) is used to select among them. The test set (open squares) enters neither training nor model selection, so the average loss on it estimates the risk of the selected hypothesis
The sizes of the three sets are design choices (Fig. 2). Under the i.i.d. assumption, the test error is an average of $|\testset|$ i.i.d. loss values, so a concentration inequality bounds its deviation from the risk: for a loss taking values in $[0,1]$, with probability at least $1-\delta$ the deviation is at most $\sqrt{\ln(2/\delta)/(2\,|\testset|)}$ (Shalev-Shwartz and Ben-David, 2014, Thm. 11.1). The test set is therefore made just large enough for the required precision; the size of the validation set is chosen by the same argument, and the training set receives every remaining data point. No general rule fixes the proportions; a typical split is 50 percent for training and 25 percent each for validation and testing (Hastie et al., 2009, Sect. 7.2).
Figure 2 of the entry testset
Figure 2: A dataset $\dataset$ split into the training set $\trainset$, the validation set $\valset$, and the test set $\testset$, with the typical proportions of Hastie et al. (2009, Sect. 7.2)
The estimate stays valid only as long as the test set is used this way. Evaluating many candidate models on $\testset$ and keeping the best turns the test set into a validation set, and its error into a validation error; data points of $\testset$ that enter the training set constitute test set contamination. In an ML competition, the labels of the test set are therefore withheld from the participants: no design choice of theirs can then depend on the test data, and the reported error keeps its meaning as a risk estimate.

Synonyms: hold-out set.

See also: training set, validation set, model selection, risk, generalization, test set contamination, data leakage, empirical risk minimization.

References

  1. Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. Cambridge Univ. Press. doi.org/10.1017/cbo9781107298019
  2. Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media. doi.org/10.1007/978-0-387-84858-7

Cite this entry

@misc{dictml_testset,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {test set},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-10-27},
  url = {https://dictionaryofml.org/terms/testset.html}
}