regularization

Jung, Alexander

doi:10.1007/978-981-92-2012-0_8-1

Dictionary of Applied Machine Learning: 114 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

regularization

Updated on 2026-10-28

Readiness 100/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Markdown Cite this entry

Regularization refers to modifications to a machine learning (ML) method that improve its generalization. Three routes are distinguished by which component of the method is modified: pruning the hypothesis space, adding a penalty term to the loss function, and augmenting the training set with perturbed copies of its data points. The three routes often yield the same learned hypothesis. For example, augmenting the training set of linear regression with Gaussian feature perturbations is equivalent to adding the ridge regression penalty term to the training error.

Figure produced by the Python demo of the entry regularization — Figure written by the entry's Python demo `regularization.py`, which recomputes what the entry states: the script block by block · open in Colab

Definition

P-overfit Training a large deep net (such as a large language model (LLM)) on a fixed training set via plain empirical risk minimization (ERM) typically leads to overfitting: the learned hypothesis $\widehat{h}$ performs well on the training set but poorly outside it. Regularization refers to modifications to a machine learning (ML) method to ensure the learned hypothesis performs nearly as well on data points outside the training set as on those in it (Goodfellow et al., 2016, Ch. 7).

P-routesFor ERM-based methods, regularization can be done in three ways:

Model pruning: shrink the hypothesis space $\mathcal{H}$ (the set of candidate hypotheses $h$) to a smaller $\mathcal{H}'$. For a parametric model, the shrinkage can be implemented via constraints on the model parameters (e.g., $w_{1} \in [0.4,0.6]$ on the weight of feature $x_{1}$ in linear regression).
Loss penalization: add a penalty term to the training error of ERM. For the squared error loss, the penalty term is an estimate of how much the risk exceeds the average loss on the training set (Hastie et al., 2009, Sect. 7.4); for other losses it controls the generalization gap without estimating it (see support vector machine (SVM)).
Data augmentation: enlarge the training set $\mathcal{D}^{(\rm train)}$ with perturbed copies of its data points (e.g., adding the realizations of independent and identically distributed (i.i.d.) random variables (RVs) to the feature vector of each data point).

Fig. 1 illustrates the three routes.

Figure 1 of the entry regularization — Figure 1: Three routes to regularization of an ERM-based method, shown for linear regression. 1) Model pruning: constraining the model parameters (here $\weight_{1} \in [0.4,0.6]$) shrinks the hypothesis space to the hypotheses inside the blue ellipse. 2) Loss penalization: the red penalty term $\regparam \regularizer{\hypothesis}$ is added to the blue training error near the bottom of the plot. 3) Data augmentation: the original data points (blue) are augmented with perturbed copies (red), displaced by an amount $\propto \sqrt{\regparam}$ (double arrow shown for the rightmost group)

These routes can yield the same learned hypothesis. As an example, consider data augmentation for linear regression that adds zero-mean perturbations with covariance matrix $\sigma^{2} \mathbf{I}$ to the feature vector of each data point. Asymptotically in the number of perturbations, the resulting learned hypothesis coincides with the one obtained from ridge regression (Bishop, 1995; Goodfellow et al., 2016, Sect. 7.5). Ridge regression adds the penalty term $\sigma^{2} \left\lVert {\mathbf w} \right\rVert_{2}^{2}$ to the training error of linear regression (Hastie et al., 2009, Sect. 3.4.1). According to Lagrangian duality, adding this penalty term is equivalent to shrinking the hypothesis space to the set of hypotheses with $\left\lVert {\mathbf w} \right\rVert_{2}^{2} \leq C$ for some constant $C$ (Bertsekas, 2016; Hastie et al., 2009, Eq. (3.42)).

P-equivThe above regularization techniques are not limited to ERM-based methods but can also be applied to other types of ML algorithms. For example, in online learning, the Follow-The-Leader (FTL) method can be regularized by adding a regularizer to the per-round objective function of FTL. The resulting Follow-The-Regularized-Leader (FTRL) method achieves lower regret than plain FTL (Hazan, 2022, Ch. 5). Another instance of regularization is the use of a prior distribution over $\mathcal{H}$ in Bayesian inference. The prior distribution acts as a regularizer: the negative logarithm of the prior distribution is a penalty term, so the maximum a posteriori estimate of the model parameters of linear regression under a Gaussian prior distribution is the ridge regression solution (Bishop, 2006, Sect. 1.2.5).

See also: overfitting, generalization, model pruning, penalty term, data augmentation, ridge regression, least absolute shrinkage and selection operator.

References

Goodfellow et al. (2016). Deep Learning. MIT Press.
Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media. doi.org/10.1007/978-0-387-84858-7
Bishop (1995). Training with Noise is Equivalent to Tikhonov Regularization. Neural Computation. doi.org/10.1162/neco.1995.7.1.108
Bertsekas (2016). Nonlinear Programming. Athena Scientific.
Hazan (2022). Introduction to Online Convex Optimization. MIT Press. sites.google.com/view/intro-oco/
Bishop (2006). Pattern Recognition and Machine Learning. Springer Science+Business Media. doi.org/10.1007/978-0-387-45528-0

Cite this entry

Published version: doi.org/10.1007/978-981-92-2012-0_8-1, in Dictionary of Applied Machine Learning (Springer Nature Singapore). Please cite that version. This page corresponds to version 1 of that entry.

@incollection{dictml_regularization,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {regularization},
  booktitle = {Dictionary of Applied Machine Learning},
  publisher = {Springer Nature Singapore},
  year = {2026},
  doi = {10.1007/978-981-92-2012-0_8-1},
  isbn = {978-981-92-2012-0},
  url = {https://doi.org/10.1007/978-981-92-2012-0_8-1}
}