# empirical risk minimization (ERM) *Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/erm.html) · [typeset PDF](https://dictionaryofml.org/terms/erm.pdf) · [LaTeX](https://dictionaryofml.org/terms/erm.tex) · [Python demo](https://dictionaryofml.org/terms/erm.py) · [notebook](https://dictionaryofml.org/terms/erm.ipynb)* Math is LaTeX with the book's macros expanded, so no notation package is needed to read it. ## Abstract Empirical risk minimization (ERM) selects a hypothesis by minimizing the average loss on a given training set $\mathcal{D}^{(\rm train)}$, the empirical risk. Machine learning methods aim at a hypothesis that incurs a small loss on any data point, and ERM replaces this unknown quantity by its average over the training set. Different machine learning methods arise from different choices of the model $\mathcal{H}$ and the loss function $L$, both of which depend on the feature space $\mathcal{X}$ and the label space $\mathcal{Y}$ of a given method. ## Definition A weather forecaster predicting tomorrow's maximum daytime temperature is useful only if its predictions are accurate on the large majority of days, not only on the days it was trained on. This forecaster is a hypothesis: it maps a day's features to a prediction of tomorrow's temperature, and its loss quantifies how far that prediction lies from the measured value. More generally, machine learning methods aim to find a hypothesis that incurs a small loss for any data point. The phrase *any data point* is made precise by a probabilistic model for the generation of data points, as follows. If data points are assumed to be drawn from a probability distribution, then the risk of a hypothesis is defined as the expectation of its loss under that probability distribution. The ideal choice is then the hypothesis with minimal risk. However, in most machine learning applications, the underlying probability distribution is not known. Instead, only a finite training set $\mathcal{D}^{(\rm train)}$ of data points is available. ERM replaces the unknown risk with its sample-average surrogate, the empirical risk on $\mathcal{D}^{(\rm train)}$, and picks the hypothesis that minimizes this surrogate (Hastie et al., 2009; Murphy, 2012; Shalev-Shwartz and Ben-David, 2014; Vapnik, 2000). For a fixed choice of loss function and hypothesis space, ERM is a map $\mathcal{A}$ that reads in a training set $\mathcal{D}^{(\rm train)} = \{{\mathbf z}^{(r)}\}_{r=1}^{m}$ of data points ${\mathbf z}^{(r)} = \left( {\mathbf x}^{(r)},y^{(r)} \right)$. The map returns a learned hypothesis $\widehat{h} = \mathcal{A}(\mathcal{D}^{(\rm train)}) \in \mathcal{H}$ that minimizes the empirical risk on $\mathcal{D}^{(\rm train)}$, $$ \widehat{h} \in \mathop{\mathrm{arg\,min}}_{h \in \mathcal{H}} \frac{1}{m} \sum_{r=1}^{m} L\left({\mathbf z}^{(r)},h \right) \text{.} $$ The hypothesis $\widehat{h}$ is chosen from the underlying hypothesis space (or model) $\mathcal{H}$. For an ERM-based method, model training is to compute $\mathcal{A}$. Different machine learning methods arise from different design choices for the two components of the map $\mathcal{A}$: the model $\mathcal{H}$ and the loss function $L$. Both depend on the nature of the data points, i.e., on the feature space $\mathcal{X}$ and the label space $\mathcal{Y}$. For example, image data points with a high-dimensional feature space and a finite label space are typically handled by an artificial neural network together with the logistic loss (Goodfellow et al., 2016, Ch. 9). Conversely, data points with few real-valued features and a real-valued label often allow for a linear model together with the squared error loss (Hastie et al., 2009, Ch. 3). Fig. 1 illustrates ERM for a linear model on data points with a single feature $x$ and label $y$. Each hypothesis is a linear map $h(x) = w_{1} x + w_{0}$ with model parameters $w_{0}, w_{1}$. ERM picks the model parameters that minimize the empirical risk on $\mathcal{D}^{(\rm train)}$. ![Figure 1 of the entry erm](https://dictionaryofml.org/terms/fig/erm-fig1.svg) *Figure 1: ERM learns a hypothesis $\widehat{h} \in \mathcal{H}$ out of a model $\mathcal{H}$ by minimizing the empirical risk (or average loss) $\tfrac{1}{m} \sum_{r=1}^{m} L\left({\mathbf z}^{(r)},h \right)$ over candidate hypotheses $h \in \mathcal{H}$ on a training set $\mathcal{D}^{(\rm train)}$. The arrows indicate the prediction errors incurred by the learned hypothesis $\widehat{h}$ (dashed line) for each data point in the training set (filled circles); the axes carry the feature $x$ and the label $y$* In practice, the map $\mathcal{A}$ is computed by an iterative optimization method. Many of these optimization methods can be represented as a fixed-point iteration that starts with an initial hypothesis $h^{(0)}$ and then repeatedly applies an update operator $\mathcal{F}$ (Bauschke and Combettes, 2011, Sect. 5.2, Cor. 27.9), $$ h^{(t+1)} = \mathcal{F}\big( h^{(t)} \big) \text{, for } t=0,1,\ldots \text{.} $$ The update operator $\mathcal{F}$ depends on the training set, the loss function, and the hypothesis space. It is chosen such that its fixed points are the stationary points of the empirical risk. These are its minimizers when the empirical risk is a convex function of the hypothesis (Boyd and Vandenberghe, 2004, Sect. 4.2.3), and need not be otherwise, e.g., for an artificial neural network. For ERM using a parametric model $\mathcal{H}$, the fixed-point iteration can be formulated directly in terms of the model parameters ${\mathbf w}$ instead of the hypothesis $h$. In particular, starting with initial model parameters ${\mathbf w}^{(0)}$, the update operator $\mathcal{F}$ is applied repeatedly to the model parameters, $$ {\mathbf w}^{(t+1)} = \mathcal{F}\big( {\mathbf w}^{(t)} \big) \text{, for } t=0,1,\ldots \text{.} $$ One example of such a fixed-point iteration is gradient descent, which updates the model parameters in the direction of the negative gradient of the empirical risk (which needs to be differentiable). Two requirements delimit where ERM applies: full feedback on the loss and access to the whole training set at once. First, ERM presupposes that the loss $L\left({\mathbf z}^{(r)},h \right)$ can be evaluated for every hypothesis $h \in \mathcal{H}$ and every training data point ${\mathbf z}^{(r)}$. This full-feedback requirement is what distinguishes ERM from reinforcement learning algorithms, which use partial feedback. In particular, at each time instant $t = 1, 2, \ldots$, a reinforcement learning algorithm chooses an action, the prediction of its current hypothesis $\widehat{h}^{(t)}$, and observes only the loss incurred by that action. It has no information about the loss that would have been incurred by any other hypothesis $h \in \mathcal{H} \setminus \{\widehat{h}^{(t)}\}$. Second, ERM uses the whole training set at once. This requirement can be relaxed: ERM can also be implemented as an online algorithm, which is useful when the data points in $\mathcal{D}^{(\rm train)}$ can only be accessed sequentially. Such sequential access arises, for instance, when memory limits prevent loading the entire training set at once. The canonical online form of ERM is the Follow-The-Leader algorithm, which, at every round, solves a partial ERM problem on all data points seen so far. Assuming that the data point ${\mathbf z}^{(t)}$ arrives at time instant $t=1,2,\ldots$, Follow-The-Leader updates the learned hypothesis $\widehat{h}^{(t)}$ by $$ \widehat{h}^{(t)} \in \mathop{\mathrm{arg\,min}}_{h \in \mathcal{H}} \frac{1}{t} \sum_{r=1}^{t} L\left({\mathbf z}^{(r)},h \right) \text{.} $$ Implementing this partial ERM separately for each time instant can become computationally expensive (and wasteful), as it does not exploit the similarity between consecutive partial ERM problems. This similarity can be exploited by online gradient descent, which performs a single gradient descent step on the new data point ${\mathbf z}^{(t)}$ at every time instant $t$ (Hazan, 2022, Sect. 3.1). **Synonyms:** empirical loss minimization. **See also:** empirical risk, risk, [loss](https://dictionaryofml.org/terms/loss.html), [training set](https://dictionaryofml.org/terms/trainset.html), [hypothesis space](https://dictionaryofml.org/terms/hypospace.html), [hypothesis](https://dictionaryofml.org/terms/hypothesis.html), [gradient descent](https://dictionaryofml.org/terms/gd.html), Follow-The-Leader, [reinforcement learning](https://dictionaryofml.org/terms/reinforcementlearning.html). ## References - Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. - Murphy (2012). Machine Learning: A Probabilistic Perspective. *MIT Press*. - Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. *Cambridge Univ. Press*. - Vapnik (2000). The Nature of Statistical Learning Theory. *Springer*. - Goodfellow et al. (2016). Deep Learning. *MIT Press*. - Bauschke and Combettes (2011). Convex Analysis and Monotone Operator Theory in Hilbert Spaces. *Springer Science+Business Media*. - Boyd and Vandenberghe (2004). Convex Optimization. *Cambridge Univ. Press*. - Hazan (2022). Introduction to Online Convex Optimization. *MIT Press*. ## Cite this entry ```bibtex @incollection{dictml_erm, author = {Jung, Alexander}, editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard}, title = {empirical risk minimization (ERM)}, booktitle = {Dictionary of Applied Machine Learning}, publisher = {Springer Nature Singapore}, year = {2026}, doi = {10.1007/978-981-92-2012-0_4-1}, isbn = {978-981-92-2012-0}, url = {https://doi.org/10.1007/978-981-92-2012-0_4-1} } ```