# regularization *Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-10-28. [HTML](https://dictionaryofml.org/terms/regularization.html) · [typeset PDF](https://dictionaryofml.org/terms/regularization.pdf) · [LaTeX](https://dictionaryofml.org/terms/regularization.tex) · [Python demo](https://dictionaryofml.org/terms/regularization.py) · [notebook](https://dictionaryofml.org/terms/regularization.ipynb)* Math is LaTeX with the book's macros expanded, so no notation package is needed to read it. ## Abstract Regularization refers to modifications to a machine learning method that improve its generalization. Three routes are distinguished by which component of the method is modified: pruning the hypothesis space, adding a penalty term to the loss function, and augmenting the training set with perturbed copies of its data points. The three routes often yield the same learned hypothesis. For example, augmenting the training set of linear regression with Gaussian feature perturbations is equivalent to adding the ridge regression penalty term to the training error. ## Definition Training a large deep net (such as a large language model) on a fixed training set via plain empirical risk minimization typically leads to overfitting: the learned hypothesis $\widehat{h}$ performs well on the training set but poorly outside it. Regularization refers to modifications to a machine learning method to ensure the learned hypothesis performs nearly as well on data points outside the training set as on those in it (Goodfellow et al., 2016, Ch. 7). For empirical risk minimization-based methods, regularization can be done in three ways: - Model pruning: shrink the hypothesis space $\mathcal{H}$ (the set of candidate hypotheses $h$) to a smaller $\mathcal{H}'$. For a parametric model, the shrinkage can be implemented via constraints on the model parameters (e.g., $w_{1} \in [0.4,0.6]$ on the weight of feature $x_{1}$ in linear regression). - Loss penalization: add a penalty term to the training error of empirical risk minimization. For the squared error loss, the penalty term is an estimate of how much the risk exceeds the average loss on the training set (Hastie et al., 2009, Sect. 7.4); for other losses it controls the generalization gap without estimating it (see support vector machine). - Data augmentation: enlarge the training set $\mathcal{D}^{(\rm train)}$ with perturbed copies of its data points (e.g., adding the realizations of independent and identically distributed RVs to the feature vector of each data point). Fig. 1 illustrates the three routes. ![Figure 1 of the entry regularization](https://dictionaryofml.org/terms/fig/regularization-fig1.svg) *Figure 1: Three routes to regularization of an empirical risk minimization-based method, shown for linear regression. 1) Model pruning: constraining the model parameters (here $w_{1} \in [0.4,0.6]$) shrinks the hypothesis space to the hypotheses inside the blue ellipse. 2) Loss penalization: the red penalty term $\alpha \mathcal{R}\big\{ h \big\}$ is added to the blue training error near the bottom of the plot. 3) Data augmentation: the original data points (blue) are augmented with perturbed copies (red), displaced by an amount $\propto \sqrt{\alpha}$ (double arrow shown for the rightmost group)* These routes can yield the same learned hypothesis. As an example, consider data augmentation for linear regression that adds zero-mean perturbations with covariance matrix $\sigma^{2} \mathbf{I}$ to the feature vector of each data point. Asymptotically in the number of perturbations, the resulting learned hypothesis coincides with the one obtained from ridge regression (Bishop, 1995; Goodfellow et al., 2016, Sect. 7.5). Ridge regression adds the penalty term $\sigma^{2} \left\lVert {\mathbf w} \right\rVert_{2}^{2}$ to the training error of linear regression (Hastie et al., 2009, Sect. 3.4.1). According to Lagrangian duality, adding this penalty term is equivalent to shrinking the hypothesis space to the set of hypotheses with $\left\lVert {\mathbf w} \right\rVert_{2}^{2} \leq C$ for some constant $C$ (Bertsekas, 2016; Hastie et al., 2009, Eq. (3.42)). The above regularization techniques are not limited to empirical risk minimization-based methods but can also be applied to other types of machine learning algorithms. For example, in online learning, the Follow-The-Leader method can be regularized by adding a regularizer to the per-round objective function of Follow-The-Leader. The resulting Follow-The-Regularized-Leader method achieves lower regret than plain Follow-The-Leader (Hazan, 2022, Ch. 5). Another instance of regularization is the use of a prior distribution over $\mathcal{H}$ in Bayesian inference. The prior distribution acts as a regularizer: the negative logarithm of the prior distribution is a penalty term, so the maximum a posteriori estimate of the model parameters of linear regression under a Gaussian prior distribution is the ridge regression solution (Bishop, 2006, Sect. 1.2.5). **See also:** [overfitting](https://dictionaryofml.org/terms/overfitting.html), [generalization](https://dictionaryofml.org/terms/generalization.html), model pruning, penalty term, [data augmentation](https://dictionaryofml.org/terms/dataaug.html), ridge regression, least absolute shrinkage and selection operator. ## References - Goodfellow et al. (2016). Deep Learning. *MIT Press*. - Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. - Bishop (1995). Training with Noise is Equivalent to Tikhonov Regularization. *Neural Computation*. - Bertsekas (2016). Nonlinear Programming. *Athena Scientific*. - Hazan (2022). Introduction to Online Convex Optimization. *MIT Press*. - Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. ## Cite this entry ```bibtex @incollection{dictml_regularization, author = {Jung, Alexander}, editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard}, title = {regularization}, booktitle = {Dictionary of Applied Machine Learning}, publisher = {Springer Nature Singapore}, year = {2026}, doi = {10.1007/978-981-92-2012-0_8-1}, isbn = {978-981-92-2012-0}, url = {https://doi.org/10.1007/978-981-92-2012-0_8-1} } ```