overfitting.py, which recomputes what the entry states: the script block by block · open in ColabDictionary of Applied Machine Learning: 114 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here
Updated on 2026-10-29
Readiness 100/100
See also empirical risk minimization generalization regularization
Overfitting is the failure mode of a machine learning (ML) method that fits its training set too closely: the learned hypothesis incurs a small empirical risk on the training set but a large risk, i.e., the expected loss on data points drawn from the underlying probability distribution. Consequently, the learned hypothesis has a large generalization gap. Validation detects overfitting by comparing the training error with the validation error obtained on a validation set. Overfitting typically arises when the hypothesis space is too large for the number of data points in the training set; the Vapnik–Chervonenkis dimension (VC dimension) and the Rademacher complexity measure that size. Regularization counteracts overfitting by pruning the model, adding a penalty term to the empirical risk, or augmenting the training set.
overfitting.py, which recomputes what the entry states: the script block by block · open in ColabAn image classifier that memorizes every photograph in its training set but misclassifies images outside its training set is overfitting. In general, consider a machine learning (ML) method that uses empirical risk minimization (ERM) to learn a hypothesis $\widehat{h} \in \mathcal{H}$ with minimum empirical risk $\widehat{L}\big(\widehat{h}|\mathcal{D}^{(\rm train)}\big)$ on a given training set $\mathcal{D}^{(\rm train)}$. The method overfits $\mathcal{D}^{(\rm train)}$ if the empirical risk $\widehat{L}\big(\widehat{h}|\mathcal{D}^{(\rm train)}\big)$ is small while the risk $\bar{L} \big( \widehat{h} \big)$, i.e., the expected loss on data points drawn, independently of $\mathcal{D}^{(\rm train)}$, from the underlying probability distribution, is large. Consequently, an overfitting ML method learns a hypothesis with a large generalization gap (Hastie et al., 2009, Sect. 7.2; Jung, 2022).
A large generalization gap alone does not imply overfitting. Consider a linear model fitted to a small training set whose labels are noisy and depend nonlinearly on the features. It can incur a large training error and a still larger risk, so the generalization gap is large while the training error is not small. That example is underfitting, the opposite failure: a hypothesis space holding no hypothesis that represents the relation between the features and the label of a data point leaves both quantities large.
Fig. 1 shows why a small training error is on its own no evidence against overfitting. Three data points lie on a straight line. None of the three curves is drawn by hand: each is a hypothesis $\widehat{h}$ delivered by ERM with the squared error loss on those same three data points. What separates them is the hypothesis space that ERM searched, and the three are nested, $\mathcal{H}^{(1)} \subset \mathcal{H}^{(2)} \subset \mathcal{H}^{(3)}$: $\mathcal{H}^{(1)}$ is the constants $h(x) = b$, $\mathcal{H}^{(2)}$ the linear model, whose hypotheses are $h(x) = w x + b$, and $\mathcal{H}^{(3)}$ all continuous functions on $\mathbb{R}$.
On $\mathcal{H}^{(1)}$, ERM has a unique solution: the constant whose value is the average of the three labels. That constant meets the middle data point and misses the other two, and no member of $\mathcal{H}^{(1)}$ does better. The hypothesis space is too small here, and the method underfits. On $\mathcal{H}^{(2)}$, the solution is unique as well and attains a training error of zero; the hypothesis space matches the relation between feature and label, and the method neither underfits nor overfits. On $\mathcal{H}^{(3)}$, every continuous function through the three data points attains a training error of zero, and there are infinitely many of them: ERM has infinitely many solutions there, and Fig. 1 draws one of them.
B-dataThe training error cannot separate that curve from the
solution on $\mathcal{H}^{(2)}$, the linear hypothesis that
passes through the three data points, and it cannot choose among the
solutions on $\mathcal{H}^{(3)}$ either. The two hypotheses agree on the training set but differ at feature values
between those of the training data points, and the
risk differs between them, since data points drawn from
the
probability distribution also fall in that region. The hypothesis space is too
large here, and the method
overfits.
B-sweepFig. 2 illustrates overfitting for polynomial regression. Each data point carries a single feature $x$ and a noisy label $y$, drawn from a probability distribution described in the caption. Polynomials of degree $p = 0, \ldots, 9$ are fitted, using ERM with the squared error loss, to training sets of $m = 5$, $10$, and $20$ data points; each panel of Fig. 2 shows the training error and the validation error for one size. The training error decreases with increasing degree in every panel: the polynomial of degree $9$ has $10$ model parameters and interpolates the $m = 10$ data points; for $m = 5$, every degree from $4$ on interpolates; for $m = 20$, no degree shown interpolates. The validation error is obtained on a shared validation set of $100$ data points. It is in contrast smallest at a moderate degree and grows by several orders of magnitude for larger degrees. The high-degree polynomials overfit the training set, and the gap between validation error and training error estimates their large generalization gap.
Comparing the three
training set sizes shows that the smaller the training set, the
lower the degree at which the validation error departs from the
training error. For $m = 5$, the validation error jumps by
nearly three orders of magnitude already at degree $4$. That is the
lowest degree whose $5$ model parameters suffice to interpolate
the training set. For $m = 20$, the validation error varies
by less than one order of magnitude across all degrees shown.
pythondemos/overfitting.py
The count is exactly right, in contrast, for the classifiers $h(x) = \operatorname{sign}\big( w_{0} + w_{1} x + \ldots + w_{p} x^{p} \big)$ obtained by thresholding the polynomials of maximum degree $p$ in a single feature. Their hypothesis space has VC dimension $p + 1$, the number of model parameters and one more than the maximum degree. The feature map $x \mapsto (x, x^{2}, \ldots, x^{p})$ turns these classifiers into linear classifiers on $\mathbb{R}^{p}$, whose hypothesis space has VC dimension $p + 1$ (Shalev-Shwartz and Ben-David, 2014, Thm. 9.3). Conversely, any $p + 1$ data points with distinct features are shattered. The classifiers realize every possible assignment of binary labels to these data points, since a polynomial of degree at most $p$ interpolates arbitrary labels on them. These thresholded classifiers concern a different case than Fig. 2, which fits polynomials of the same degrees to a regression task.
Both measures enter the upper bounds on the generalization gap, which grow with them and shrink with the number of data points in the training set (Shalev-Shwartz and Ben-David, 2014, Sect. 26.1; Vapnik, 2000, Sect. 3.7).
B-regRegularization counteracts overfitting in three elementary forms: pruning the model, adding a penalty term to the empirical risk, or augmenting the training set via data augmentation.
See also: empirical risk minimization, generalization gap, generalization, regularization, underfitting, validation, Vapnik–Chervonenkis dimension.
@misc{dictml_overfitting,
author = {Jung, Alexander},
editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
title = {overfitting},
howpublished = {Dictionary of Applied Machine Learning (course edition)},
year = {2026},
doi = {10.5281/zenodo.21569296},
note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-04},
url = {https://dictionaryofml.org/terms/overfitting.html}
}