modelsel.py, which recomputes what the entry states: the script block by block · open in ColabDictionary of Applied Machine Learning: 115 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here
Updated on 2026-11-10
Readiness 100/100
See also validation error $k$-fold cross-validation training set validation set test set hyperparameter hypothesis space overfitting
Model selection chooses among several alternative machine learning (ML) methods by comparing their validation errors. Each ML method learns a hypothesis via a training method such as empirical risk minimization (ERM). A core design choice of a ML method is the underlying hypothesis space, or model. The effective hypothesis space can also depend on hyperparameters of the optimization algorithms used to solve ERM. The training error of an ML method cannot serve as the criterion for model selection, as it always chooses the largest of nested hypothesis spaces. The validation error can be obtained from the average loss on a single validation set, or via $k$-fold cross-validation ($k$-fold CV). The ultimately chosen hypothesis must be assessed on data points that have been used neither for learning any individual hypothesis nor for choosing among them.
modelsel.py, which recomputes what the entry states: the script block by block · open in ColabB-trainerrFig. 1 depicts data points collected by a weather station at Krems an der Donau. Each data point represents a day, with the morning minimum temperature as feature $x$ and the maximum daytime temperature as label $y$. The figure also shows two hypothesis maps, each learned by a machine learning (ML) method via empirical risk minimization (ERM) from the training set $\mathcal{D}^{(\rm train)}$ of six data points. One is a linear hypothesis $\widehat{h}^{(1)}$, learned over a linear model. The other is a degree-five polynomial $\widehat{h}^{(2)}$, learned over the larger hypothesis space of polynomial regression.
B-valerrThe polynomial delivers perfect predictions on every data point of $\mathcal{D}^{(\rm train)}$, so its training error, the average squared error loss over $\mathcal{D}^{(\rm train)}$, is zero. The linear hypothesis misses each of them and its training error is $0.47$. The training error therefore ranks the polynomial first, and following that ranking would pick the candidate that predicts worst on a data point outside $\mathcal{D}^{(\rm train)}$. Model selection ranks the two instead by the validation error on a validation set $\mathcal{D}^{(\rm val)}$ that ERM did not use. On the fourteen validation data points of Fig. 1, the average squared error loss is $2065$ for the polynomial and $7.06$ for the linear hypothesis, so model selection picks the linear hypothesis $\widehat{h}^{(1)}$.
B-testreuseThe validation error of the selected hypothesis is biased downward, since the selection favored the candidate with the smallest one. Model selection therefore leaves a test set $\mathcal{D}^{(\rm test)}$ untouched for estimating the risk of the selected hypothesis.
pythondemos/validation.py
The training error of the learned hypotheses $\widehat{h}^{(c)}$ cannot serve as the criterion for choosing between them. For nested candidates $\mathcal{H}^{(1)} \subseteq \cdots \subseteq \mathcal{H}^{(C)}$ it always chooses the largest hypothesis space, since the minimum of the training error over a superset is no larger, even when that results in overfitting, as the degree-five polynomial of Fig. 1 does (Hastie et al., 2009, Sect. 7.2, Fig. 7.1; Shalev-Shwartz and Ben-David, 2014, Sect. 11.2.3). Model selection therefore follows model training, e.g., via ERM, and model validation, e.g., via $k$-fold cross-validation ($k$-fold CV).
Algorithm 1 summarizes a complete ML
workflow, consisting of model training, model
validation and model selection.
The final hypothesis $\widehat{h}^{(c^{\star})}$ delivered by Algorithm 1 must be evaluated on a test set. A test set consists of data points that have been used neither for learning any individual $\widehat{h}^{(c)}$ nor for choosing between them. Under an independent and identically distributed assumption (i.i.d. assumption) for the data generation, the average loss on $\mathcal{D}^{(\rm test)}$ is an unbiased estimator of the risk of $\widehat{h}^{(c^{\star})}$: the data points of $\mathcal{D}^{(\rm test)}$ are independent of how $\widehat{h}^{(c^{\star})}$ was obtained (Shalev-Shwartz and Ben-David, 2014, Sect. 11.2.1), so each loss has the risk as its expectation and so has their average (Bertsekas and Tsitsiklis, 2008, Sect. 9.1). If $\mathcal{D}^{(\rm train)}$ and $\mathcal{D}^{(\rm val)}$ are modelled as random, so is $\widehat{h}^{(c^{\star})}$, and with it the risk being estimated: the expectation is then conditional on $\widehat{h}^{(c^{\star})}$, over $\mathcal{D}^{(\rm test)}$ alone.
See also: validation, validation error, $k$-fold cross-validation, training set, validation set, test set, hyperparameter, hypothesis space, overfitting.
@misc{dictml_modelsel,
author = {Jung, Alexander},
editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
title = {model selection},
howpublished = {Dictionary of Applied Machine Learning (course edition)},
year = {2026},
doi = {10.5281/zenodo.21569296},
note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-13},
url = {https://dictionaryofml.org/terms/modelsel.html}
}