Dictionary of Applied Machine Learning

overfitting

Typeset PDF version — the authoritative form of this entry

Overfitting is the failure mode of a machine learning (ML) method that fits its training set too closely: the learned hypothesis incurs a small empirical risk on the training set but a large risk, i.e., a large expected loss on data points outside the training set. Equivalently, the learned hypothesis has a large generalization gap. Overfitting typically arises when the hypothesis space is too large relative to the number of data points in the training set. Regularization counteracts overfitting by pruning the hypothesis space, augmenting the training set, or adding a penalty term to the empirical risk. Validation detects overfitting by comparing the training error with the validation error obtained on a validation set.

Definition

An image classifier that memorizes every photograph in its training set but fails to correctly classify previously unseen images is overfitting. In general, consider a machine learning (ML) method that uses empirical risk minimization (ERM) to learn a hypothesis $\learnthypothesis \in \hypospace$ with minimum empirical risk $\emprisk{\learnthypothesis}{\trainset}$ on a given training set $\trainset$. The method overfits $\trainset$ if the empirical risk $\emprisk{\learnthypothesis}{\trainset}$ is small while the risk $\risk{\learnthypothesis}$, i.e., the expected loss incurred on data points outside $\trainset$, is large. In other words, an overfitting ML method learns a hypothesis with a large generalization gap (Hastie et al., 2009; Jung, 2022).

Overfitting typically occurs when the hypothesis space $\hypospace$ is too large relative to the number of data points in the training set. Regularization counteracts overfitting in three elementary forms: pruning the model, augmenting the training set via data augmentation, or adding a penalty term to the empirical risk. Validation detects overfitting: an ML method that overfits delivers a small training error but a large validation error on a validation set.

Fig. 1 illustrates overfitting for polynomial regression. Polynomials of degree $0, \ldots, 9$ are fitted, using ERM with the squared error loss, to a training set of $\samplesize = 10$ data points. The training error decreases with increasing degree. The polynomial of degree $9$ has $10$ model parameters and interpolates the $\samplesize = 10$ data points. The validation error, in contrast, is smallest for degree $3$ and grows by several orders of magnitude for larger degrees: the high-degree polynomials overfit the training set. % Data generated by pythondemos/overfitting.py

Figure 1 of the entry overfitting
Figure 1: Training error and validation error of polynomial regression over the polynomial degree. The training set consists of $\samplesize = 10$ data points with feature $\feature$ drawn uniformly from $[0,1]$ and label $\truelabel = \sin( 2 \pi \feature ) + \varepsilon$ with additive Gaussian noise $\varepsilon$. The validation set consists of $100$ data points drawn from the same probability distribution. Data generated by pythondemos/overfitting.py
See also: ERM, generalization gap, generalization, regularization, underfitting, validation.

References

  1. Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media.
  2. Jung (2022). Machine Learning: The Basics. Springer Nature.

Cite this entry

@misc{dictml_overfitting,
  author = {Jung, Alexander},
  title = {overfitting},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-08-06},
  url = {https://dictionaryofml.org/terms/overfitting.html}
}