erm.py, which recomputes what the entry states: the script block by block · open in ColabDictionary of Applied Machine Learning: 115 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here
Updated on 2026-11-08
Readiness 100/100
See also loss training set hypothesis space hypothesis gradient descent reinforcement learning
Empirical risk minimization (ERM) selects a hypothesis by minimizing the average loss on a given training set $\mathcal{D}^{(\rm train)}$, the empirical risk. Machine learning (ML) methods aim at a hypothesis that incurs a small loss on any data point, and ERM replaces this unknown quantity by its average over the training set. Different ML methods arise from different choices of the model $\mathcal{H}$ and the loss function $L$, both of which depend on the feature space $\mathcal{X}$ and the label space $\mathcal{Y}$ of a given method.
erm.py, which recomputes what the entry states: the script block by block · open in ColabA weather forecaster predicting tomorrow's maximum daytime temperature is useful only if its predictions are accurate on the large majority of days, not only on the days it was trained on. This forecaster is a hypothesis: it maps a day's features to a prediction of tomorrow's temperature, and its loss quantifies how far that prediction lies from the measured value. More generally, machine learning (ML) methods aim to find a hypothesis that incurs a small loss for any data point. The phrase any data point is made precise by a probabilistic model for the generation of data points, as follows.
P-surrogateIf data points are assumed to be drawn from a probability distribution, then the risk of a hypothesis is defined as the expectation of its loss under that probability distribution. The ideal choice is then the hypothesis with minimal risk. However, in most ML applications, the underlying probability distribution is not known. Instead, only a finite training set $\mathcal{D}^{(\rm train)}$ of data points is available. ERM replaces the unknown risk with its sample-average surrogate, the empirical risk on $\mathcal{D}^{(\rm train)}$, and picks the hypothesis that minimizes this surrogate (Hastie et al., 2009; Murphy, 2012; Shalev-Shwartz and Ben-David, 2014; Vapnik, 2000).
For a fixed choice of loss function and hypothesis space, ERM is a map $\mathcal{A}$ that reads in a training set $\mathcal{D}^{(\rm train)} = \{{\mathbf z}^{(r)}\}_{r=1}^{m}$ of data points ${\mathbf z}^{(r)} = \left( {\mathbf x}^{(r)},y^{(r)} \right)$. The map returns a learned hypothesis $\widehat{h} = \mathcal{A}(\mathcal{D}^{(\rm train)}) \in \mathcal{H}$ that minimizes the empirical risk on $\mathcal{D}^{(\rm train)}$, \[ \widehat{h} \in \mathop{\mathrm{arg\,min}}_{h \in \mathcal{H}} \frac{1}{m} \sum_{r=1}^{m} L\left({\mathbf z}^{(r)},h \right) \text{.} \] The hypothesis $\widehat{h}$ is chosen from the underlying hypothesis space (or model) $\mathcal{H}$. For an ERM-based method, model training is to compute $\mathcal{A}$.
P-mapDifferent ML methods arise from different design choices for the two components of the map $\mathcal{A}$: the model $\mathcal{H}$ and the loss function $L$. Both depend on the nature of the data points, i.e., on the feature space $\mathcal{X}$ and the label space $\mathcal{Y}$. For example, image data points with a high-dimensional feature space and a finite label space are typically handled by an artificial neural network (ANN) together with the logistic loss (Goodfellow et al., 2016, Ch. 9). Conversely, data points with few real-valued features and a real-valued label often allow for a linear model together with the squared error loss (Hastie et al., 2009, Ch. 3).
P-fixedpointFig. 1 illustrates ERM for a linear model
on data points with a single feature $x$
and label $y$. Each hypothesis is a
linear map
$h(x) = w_{1} x + w_{0}$
with model parameters $w_{0}, w_{1}$. ERM
picks the model parameters that minimize the empirical risk
on $\mathcal{D}^{(\rm train)}$.
For ERM using a parametric model $\mathcal{H}$, the fixed-point iteration can be formulated directly in terms of the model parameters ${\mathbf w}$ instead of the hypothesis $h$. In particular, starting with initial model parameters ${\mathbf w}^{(0)}$, the update operator $\mathcal{F}$ is applied repeatedly to the model parameters, \[{\mathbf w}^{(t+1)} = \mathcal{F}\big( {\mathbf w}^{(t)} \big) \text{, for } t=0,1,\ldots \text{.}\] One example of such a fixed-point iteration is gradient descent (GD), which updates the model parameters in the direction of the negative gradient of the empirical risk (which needs to be differentiable).
Two requirements delimit where ERM applies: full feedback on the loss and access to the whole training set at once. First, ERM presupposes that the loss $L\left({\mathbf z}^{(r)},h \right)$ can be evaluated for every hypothesis $h \in \mathcal{H}$ and every training data point ${\mathbf z}^{(r)}$. This full-feedback requirement is what distinguishes ERM from reinforcement learning (RL) algorithms, which use partial feedback. In particular, at each time instant $t = 1, 2, \ldots$, an RL algorithm chooses an action, the prediction of its current hypothesis $\widehat{h}^{(t)}$, and observes only the loss incurred by that action. It has no information about the loss that would have been incurred by any other hypothesis $h \in \mathcal{H} \setminus \{\widehat{h}^{(t)}\}$.
Second, ERM uses the whole training set at once. This requirement can be relaxed: ERM can also be implemented as an online algorithm, which is useful when the data points in $\mathcal{D}^{(\rm train)}$ can only be accessed sequentially. Such sequential access arises, for instance, when memory limits prevent loading the entire training set at once. The canonical online form of ERM is the Follow-The-Leader (FTL) algorithm, which, at every round, solves a partial ERM problem on all data points seen so far. Assuming that the data point ${\mathbf z}^{(t)}$ arrives at time instant $t=1,2,\ldots$, FTL updates the learned hypothesis $\widehat{h}^{(t)}$ by \[ \widehat{h}^{(t)} \in \mathop{\mathrm{arg\,min}}_{h \in \mathcal{H}} \frac{1}{t} \sum_{r=1}^{t} L\left({\mathbf z}^{(r)},h \right) \text{.}\] Implementing this partial ERM separately for each time instant can become computationally expensive (and wasteful), as it does not exploit the similarity between consecutive partial ERM problems. This similarity can be exploited by online gradient descent (online GD), which performs a single GD step on the new data point ${\mathbf z}^{(t)}$ at every time instant $t$ (Hazan, 2022, Sect. 3.1).
Synonyms: empirical loss minimization.
See also: empirical risk, risk, loss, training set, hypothesis space, hypothesis, gradient descent, Follow-The-Leader, reinforcement learning.
Published version: doi.org/10.1007/978-981-92-2012-0_4-1, in Dictionary of Applied Machine Learning (Springer Nature Singapore). Please cite that version. This page corresponds to version 1 of that entry.
@incollection{dictml_erm,
author = {Jung, Alexander},
editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
title = {empirical risk minimization (ERM)},
booktitle = {Dictionary of Applied Machine Learning},
publisher = {Springer Nature Singapore},
year = {2026},
doi = {10.1007/978-981-92-2012-0_4-1},
isbn = {978-981-92-2012-0},
url = {https://doi.org/10.1007/978-981-92-2012-0_4-1}
}