Dictionary of Applied Machine Learning: 114 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

generalization

Updated on 2026-10-29

Readiness 100/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Markdown Cite this entry

See also independent and identically distributed assumption training set overfitting regularization

Generalization is the ability of a machine learning (ML) method to make accurate predictions on data points that have not been used during training. Many ML methods implement training via empirical risk minimization (ERM), i.e., learning a hypothesis $\widehat{h}$ that minimizes the empirical risk on the training set $\mathcal{D}^{(\rm train)}$. The learned $\widehat{h}$ is then used to compute predictions on new data points outside $\mathcal{D}^{(\rm train)}$. A probabilistic model for data generation quantifies the prediction errors beyond $\mathcal{D}^{(\rm train)}$ via the risk of $\widehat{h}$, i.e., the expected loss incurred on a new randomly chosen data point. The difference between the risk and the empirical risk is the generalization gap. The stability of the ML method bounds the expected generalization gap without reference to the size of the hypothesis space. A complementary deterministic view characterizes generalization via the robustness of $\widehat{h}$: its prediction changes by at most a bounded multiple of a perturbation of the features of a data point.

Figure produced by the Python demo of the entry generalization
Figure written by the entry's Python demo generalization.py, which recomputes what the entry states: the script block by block · open in Colab

Definition

P-goalA weather forecaster trained on past data is useful only if its predictions are accurate on data points that did not belong to the training set. Generalization refers to the ability of a learned hypothesis $\widehat{h}$ to make similarly accurate predictions on data points that have not been used during training. It is the central goal of machine learning (ML): learning patterns in the features and labels of data points that extend beyond the training set.

P-ermThe canonical learning technique is empirical risk minimization (ERM): the machine learning system (ML system) obtains $\widehat{h} \in \mathcal{H}$ by minimizing the empirical risk on the training set $\mathcal{D}^{(\rm train)}$. A low empirical risk on $\mathcal{D}^{(\rm train)}$ does not guarantee accurate predictions on data points outside $\mathcal{D}^{(\rm train)}$. The same question arises for online learning and Bayesian inference, where the training set is replaced by the data points processed so far or by a prior.

P-stableStudying generalization mathematically requires formalizing the notion of data points outside the training set. A widely used approach is to assume a probabilistic model for the generation of data points, such as the independent and identically distributed assumption (i.i.d. assumption). Here, data points are interpreted as independent random variables (RVs) with an identical, fixed but unknown, probability distribution $\mathbb{P}$. Fig. 1 sketches this setup: the shaded ellipse depicts the region of typical data points under $\mathbb{P}$, and the two large filled circles are the data points of the training set that ERM fits with the learned hypothesis $\widehat{h}$, and the arrows are small perturbations of them. The risk of $\widehat{h}$ is the expected loss: \[ \bar{L} \big( \widehat{h} \big) = \mathbb{E} \left\{ L\left({\mathbf z},\widehat{h} \right) \right\}. \] The difference $\bar{L} \big( \widehat{h} \big) - \widehat{L}\big(\widehat{h}|\mathcal{D}^{(\rm train)}\big)$ is the generalization gap.

Figure 1 of the entry generalization
Figure 1: Two data points $\datapoint^{(1)}, \datapoint^{(2)}$ used as a training set to learn a hypothesis $\learnthypothesis$ via ERM. The shaded ellipse marks the region of typical data points under an assumed probability distribution $\probdist$
For any fixed hypothesis $h \in \mathcal{H}$, $\bar{L} \big( h \big)$ is a deterministic number that depends only on $h$ and $\mathbb{P}$. In contrast, the empirical risk $\widehat{L}\big(h|\mathcal{D}^{(\rm train)}\big)$ is an RV that depends on the training set $\mathcal{D}^{(\rm train)}$ of size $m$, which, in turn, is a realization of an independent and identically distributed (i.i.d.) sequence of RVs ${\mathbf z}^{(1)}, \ldots, {\mathbf z}^{(m)} \sim \mathbb{P}$. Bounding the generalization gap therefore means controlling the probability of the (undesirable) event \[ \mathcal{A}^{(h)} := \left\{ \mathcal{D}^{(\rm train)} : \big| \widehat{L}\big(h|\mathcal{D}^{(\rm train)}\big) - \bar{L} \big( h \big) \big| > \varepsilon \right\}, \] where $\varepsilon > 0$ is a user-specified tolerance for the generalization gap. Each $h \in \mathcal{H}$ defines a different event $\mathcal{A}^{(h)}$ (see Fig. 2), and generalization requires low probability of $\mathcal{A}^{(h)}$ uniformly over $\mathcal{H}$ (Shalev-Shwartz and Ben-David, 2014, Sect. 4.1; Vapnik, 2000). Fig. 2 draws these events for a feature space and a label space that are both the real line, as subsets of the sample space of all training sets of size $m$.
Figure 2 of the entry generalization
Figure 2: Consider an ML application with feature space $\featurespace=\reals$ and label space $\labelspace=\reals$. A training set of size $\samplesize$ is a point in the sample space $(\reals \times \reals)^{\samplesize}$ of all possible training sets of size $\samplesize$ (the outer region). Each hypothesis $\hypothesis \in \hypospace$ defines an event $\genericevent^{(\hypothesis)} \subseteq (\reals \times \reals)^{\samplesize}$ (an inner region) collecting those training sets on which $\hypothesis$ fails to generalize. Ensuring that the ML method is likely to generalize requires the union $\bigcup_{\hypothesis \in \hypospace} \genericevent^{(\hypothesis)}$ to have low probability, not just each individual $\genericevent^{(\hypothesis)}$
Controlling every event $\mathcal{A}^{(h)}$ uniformly over $\mathcal{H}$ is not the only probabilistic route. If the ML method is stable, i.e., replacing a single data point of the training set changes the loss of the learned hypothesis only slightly on average, then the expected generalization gap equals that average change, regardless of the size of the hypothesis space (see stability; Shalev-Shwartz and Ben-David, 2014, Thm. 13.2).

Probability theory is not the only route to generalization. A complementary deterministic approach uses robustness: a good hypothesis $\widehat{h}$ should change its prediction $\widehat{h}({\mathbf x})$ by at most a bounded multiple of the perturbation of the feature vector ${\mathbf x}$ of a data point ${\mathbf z}$, i.e., it should be Lipschitz. For example, an object detection system trained on smartphone photos should still detect the object if a single pixel is changed. Trained deep nets have been shown to lack this property (Su et al., 2019). The system should also deliver the same result if the object in the image is slightly translated or deformed (Mallat, 2016, Sect. 3). The arrows in Fig. 1 indicate small perturbations of each data point in the training set of an ML method.

See also: generalization gap, risk, empirical risk, independent and identically distributed assumption, training set, overfitting, regularization, stability.

References

  1. Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. Cambridge Univ. Press. doi.org/10.1017/cbo9781107298019
  2. Vapnik (2000). The Nature of Statistical Learning Theory. Springer. doi.org/10.1007/978-1-4757-3264-1
  3. Su et al. (2019). One Pixel Attack for Fooling Deep Neural Networks. IEEE Trans. Evol. Comput.. doi.org/10.1109/TEVC.2019.2890858
  4. Mallat (2016). Understanding deep convolutional networks. Philos. Trans. Roy. Soc. A. doi.org/10.1098/rsta.2015.0203

Cite this entry

Published version: doi.org/10.1007/978-981-92-2012-0_7-1, in Dictionary of Applied Machine Learning (Springer Nature Singapore). Please cite that version. This page corresponds to version 1 of that entry.

@incollection{dictml_generalization,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {generalization},
  booktitle = {Dictionary of Applied Machine Learning},
  publisher = {Springer Nature Singapore},
  year = {2026},
  doi = {10.1007/978-981-92-2012-0_7-1},
  isbn = {978-981-92-2012-0},
  url = {https://doi.org/10.1007/978-981-92-2012-0_7-1}
}