# model selection

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-10. [HTML](https://dictionaryofml.org/terms/modelsel.html) · [typeset PDF](https://dictionaryofml.org/terms/modelsel.pdf) · [LaTeX](https://dictionaryofml.org/terms/modelsel.tex) · [Python demo](https://dictionaryofml.org/terms/modelsel.py) · [notebook](https://dictionaryofml.org/terms/modelsel.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Model selection chooses among several alternative machine learning methods by comparing their validation errors. Each machine learning method learns a hypothesis via a training method such as empirical risk minimization. A core design choice of a machine learning method is the underlying hypothesis space, or model. The effective hypothesis space can also depend on hyperparameters of the optimization algorithms used to solve empirical risk minimization. The training error of a machine learning method cannot serve as the criterion for model selection, as it always chooses the largest of nested hypothesis spaces. The validation error can be obtained from the average loss on a single validation set, or via k-fold cross-validation. The ultimately chosen hypothesis must be assessed on data points that have been used neither for learning any individual hypothesis nor for choosing among them.

## Definition

Fig. 1 depicts data points collected by a weather station at Krems an der Donau. Each data point represents a day, with the morning minimum temperature as feature $x$ and the maximum daytime temperature as label $y$. The figure also shows two hypothesis maps, each learned by a machine learning method via empirical risk minimization from the training set $\mathcal{D}^{(\rm train)}$ of six data points. One is a linear hypothesis $\widehat{h}^{(1)}$, learned over a linear model. The other is a degree-five polynomial $\widehat{h}^{(2)}$, learned over the larger hypothesis space of polynomial regression.

The polynomial delivers perfect predictions on every data point of $\mathcal{D}^{(\rm train)}$, so its training error, the average squared error loss over $\mathcal{D}^{(\rm train)}$, is zero. The linear hypothesis misses each of them and its training error is $0.47$. The training error therefore ranks the polynomial first, and following that ranking would pick the candidate that predicts worst on a data point outside $\mathcal{D}^{(\rm train)}$. Model selection ranks the two instead by the validation error on a validation set $\mathcal{D}^{(\rm val)}$ that empirical risk minimization did not use. On the fourteen validation data points of Fig. 1, the average squared error loss is $2065$ for the polynomial and $7.06$ for the linear hypothesis, so model selection picks the linear hypothesis $\widehat{h}^{(1)}$.

The validation error of the selected hypothesis is biased downward, since the selection favored the candidate with the smallest one. Model selection therefore leaves a test set $\mathcal{D}^{(\rm test)}$ untouched for estimating the risk of the selected hypothesis.

![Figure 1 of the entry modelsel](https://dictionaryofml.org/terms/fig/modelsel-fig1.svg)

*Figure 1: Model selection between two machine learning methods that learn a hypothesis via empirical risk minimization from the same six data points (filled circles). One method uses degree-five polynomials as hypothesis space, the other a linear model. The degree-five polynomial $\widehat{h}^{(2)}$ (dashed curve) delivers perfect predictions on every data point of the training set, so its training error is zero against $0.47$ for the linear hypothesis $\widehat{h}^{(1)}$ (solid curve). Averaged over the fourteen validation data points (open triangles), the squared error loss is $2065$ for the polynomial and $7.06$ for the linear hypothesis, so the validation error selects the linear hypothesis $\widehat{h}^{(1)}$. The six data points of the test set (open squares) are held out from both empirical risk minimization and the model selection, so the average squared error loss on them, $1.83$, estimates the risk of the selected hypothesis. Data generated by pythondemos/validation.py*

Model selection chooses among several alternative machine learning methods by comparing their validation errors (Shalev-Shwartz and Ben-David, 2014, Sect. 11.2.2). Consider $C$ candidates $\widehat{h}^{(c)}$, $c = 1, \ldots, C$, obtained from empirical risk minimization with a different effective hypothesis space, or model, $\mathcal{H}^{(c)}$. The effective hypothesis space can also depend on hyperparameters of the optimization algorithm used to solve empirical risk minimization. Typical hyperparameters include the learning rate of a gradient-based method, the strength $\alpha$ of a regularization penalty term, and the depth of a decision tree (Goodfellow et al., 2016, Sect. 11.4).

The training error of the learned hypotheses $\widehat{h}^{(c)}$ cannot serve as the criterion for choosing between them. For nested candidates $\mathcal{H}^{(1)} \subseteq \cdots \subseteq \mathcal{H}^{(C)}$ it always chooses the largest hypothesis space, since the minimum of the training error over a superset is no larger, even when that results in overfitting, as the degree-five polynomial of Fig. 1 does (Hastie et al., 2009, Sect. 7.2, Fig. 7.1; Shalev-Shwartz and Ben-David, 2014, Sect. 11.2.3). Model selection therefore follows model training, e.g., via empirical risk minimization, and model validation, e.g., via k-fold cross-validation.

Algorithm 1 summarizes a complete machine learning workflow, consisting of model training, model validation and model selection.

![Algorithm 1 of the entry modelsel](https://dictionaryofml.org/terms/fig/modelsel-alg1.svg)

*Algorithm 1: Model selection*

The two loss functions used in Algorithm 1 for training and validation can be different: empirical risk minimization needs one that its optimization method can handle, while validation can use the loss function that matters for the application, such as the 0/1 loss. When the training set is too small to spare a validation set, steps 2 and 3 of Algorithm 1 can be replaced by k-fold cross-validation (Hastie et al., 2009, Sect. 7.10).

The final hypothesis $\widehat{h}^{(c^{\star})}$ delivered by Algorithm 1 must be evaluated on a test set. A test set consists of data points that have been used neither for learning any individual $\widehat{h}^{(c)}$ nor for choosing between them. Under an independent and identically distributed assumption for the data generation, the average loss on $\mathcal{D}^{(\rm test)}$ is an unbiased estimator of the risk of $\widehat{h}^{(c^{\star})}$: the data points of $\mathcal{D}^{(\rm test)}$ are independent of how $\widehat{h}^{(c^{\star})}$ was obtained (Shalev-Shwartz and Ben-David, 2014, Sect. 11.2.1), so each loss has the risk as its expectation and so has their average (Bertsekas and Tsitsiklis, 2008, Sect. 9.1). If $\mathcal{D}^{(\rm train)}$ and $\mathcal{D}^{(\rm val)}$ are modelled as random, so is $\widehat{h}^{(c^{\star})}$, and with it the risk being estimated: the expectation is then conditional on $\widehat{h}^{(c^{\star})}$, over $\mathcal{D}^{(\rm test)}$ alone.

**See also:** validation, [validation error](https://dictionaryofml.org/terms/valerr.html), [$k$-fold cross-validation](https://dictionaryofml.org/terms/kfoldcv.html), [training set](https://dictionaryofml.org/terms/trainset.html), [validation set](https://dictionaryofml.org/terms/valset.html), [test set](https://dictionaryofml.org/terms/testset.html), [hyperparameter](https://dictionaryofml.org/terms/hyperparameter.html), [hypothesis space](https://dictionaryofml.org/terms/hypospace.html), [overfitting](https://dictionaryofml.org/terms/overfitting.html).

## References

- Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. <https://doi.org/10.1017/cbo9781107298019>
- Goodfellow et al. (2016). Deep Learning. *MIT Press*.
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Bertsekas and Tsitsiklis (2008). Introduction to Probability. *Athena Scientific*.

## Cite this entry

```bibtex
@misc{dictml_modelsel,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {model selection},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/modelsel.html}
}
```