# ensemble

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-07. [HTML](https://dictionaryofml.org/terms/ensemble.html) · [typeset PDF](https://dictionaryofml.org/terms/ensemble.pdf) · [LaTeX](https://dictionaryofml.org/terms/ensemble.tex) · [Python demo](https://dictionaryofml.org/terms/ensemble.py) · [notebook](https://dictionaryofml.org/terms/ensemble.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

An ensemble method combines several machine learning methods, each referred to as a base learner. Ensemble methods differ in how they construct the base learners and in how they combine their results. Bagging trains each base learner on a resampled copy of the training set. Boosting runs them sequentially, each one fitted to the errors of the previous ones. Stacking uses base learners to train different models on the same training set and learns an aggregation rule. For the squared error loss the gain is exact: the error of the average prediction is the average error of the base learners minus their average disagreement, so base learners that disagree are what makes an ensemble worth forming.

## Definition

Predicting the maximum temperature of a day at the weather station Krems an der Donau from the minimum temperature of the same day can be done with a linear model, with a decision tree, or with an artificial neural network. Fig. 1 shows all three learned from the $366$ days of $2020$. None of them is the best everywhere: the linear hypothesis misses the curvature at the cold end, the decision tree answers with a staircase, and the artificial neural network bends where the other two cannot. Model selection would keep one of them and discard the rest. Combining these base learners can do better than choosing one of them.

![Figure 1 of the entry ensemble](https://dictionaryofml.org/terms/fig/ensemble-fig1.svg)

*Figure 1: Three base learners on one task: the maximum temperature of a day at Krems an der Donau against the minimum temperature of the same day. The dots are $20$ of the $366$ days of $2020$, every $19$th day. The linear model (dash-dotted curve), the decision tree (dotted curve) and the artificial neural network (dashed curve) are each learned from the same $244$ training days. On the $122$ held-back days their average squared errors are $26.20$, $27.09$, and $26.13$, so none of them is the best; the average of the three (solid curve) reaches $25.95$. Data generated by pythondemos/ensemble.py*

Combining several machine learning methods into one predictor is an ensemble method: each combined method is a base learner (Hastie et al., 2009, Ch. 16). The base learners can be empirical risk minimization-based, using different choices for the loss, model, and training set. Of $M$ base learners, base learner $\ell$ learns a hypothesis $\widehat{h}^{(\ell)}$ from its hypothesis space $\mathcal{H}^{(\ell)}$ and its, possibly perturbed, training set $\widetilde{\mathcal{D}}^{(\ell)}$, and an aggregation rule $\phi_{\text{agg}}$ combines the outputs of these hypotheses into the final prediction. The aggregation can amount to averaging (in regression), as in Fig. 1, or to a majority vote (in classification); Fig. 2 shows the general shape.

![Figure 2 of the entry ensemble](https://dictionaryofml.org/terms/fig/ensemble-fig2.svg)

*Figure 2: Generic ensemble with three base learners: base learner $\ell \in \{1, 2, 3\}$ uses empirical risk minimization to learn $\widehat{h}^{(\ell)} \in \mathcal{H}^{(\ell)}$ from the perturbed training set $\widetilde{\mathcal{D}}^{(\ell)}$. A base learner might also use the output of other base learners (dashed arrows). The final hypothesis $\widehat{h}$ is obtained by aggregating the hypotheses of the base learners via an aggregation rule $\phi_{\text{agg}}$*

What the combination gains is exactly the disagreement among the base learners. For the squared error loss, the error of the average prediction is the average error of the base learners minus their average disagreement, where the disagreement of a base learner is the squared difference between its own prediction and the average one (Krogh and Vedelsby, 1994). This is an identity, not an approximation: on the held-back days of Fig. 1 the average member error $26.47$ minus the average disagreement $0.52$ is the $25.95$ that the average prediction actually incurs. Two consequences follow. The average is never worse than its base learners are on average, since a disagreement cannot be negative. And base learners that agree everywhere gain nothing, which is why they are trained on different models or on different data points. Outperforming the *best* single base learner is a stronger claim, and one the identity does not guarantee: over $50$ repetitions on random halves of the training set, the average did so $22$ times.

The same point in terms of variance: consider $M$ base learners whose predictions are identically distributed, each with variance $\sigma^{2}$ and pairwise correlation $\rho$. The average of these predictions has variance (Hastie et al., 2009, Sect. 15.2)

$$
\rho \sigma^{2} + \frac{1-\rho}{M}\,\sigma^{2}\text{.}
$$

The second of these two summands vanishes as $M$ grows, while the first, set by the correlation between the base learners, limits what averaging can achieve.

Ensemble methods differ in how they construct the base learners and in how they combine their results. Bagging trains each base learner on a random resample of the training set and averages them; it reduces the variance of an unstable base learner such as a deep decision tree, while the systematic part of its error, the bias, is left unchanged (Hastie et al., 2009, Sect. 8.7). A random forest is bagging of decision trees with, in addition, a random choice of the features considered at each split, which lowers the correlation between the trees (Breiman, 2001, Sect. 3; Hastie et al., 2009, Sect. 15.2).

Boosting runs the base learners sequentially, each one fitted to the errors of the previous ones. The number of rounds is a hyperparameter: the training error keeps decreasing while the validation error eventually rises, so boosting overfits when run for too long (Hastie et al., 2009, Sect. 10.12). Stacking trains the base learners on the same training set with different models and learns the aggregation rule itself by a second machine learning method. The second method is trained on predictions for data points that did not belong to the training sets of the base learners. Otherwise the base learners that fit the training set best are favored regardless of how they generalize (Hastie et al., 2009, Sect. 8.8).

**See also:** [bagging](https://dictionaryofml.org/terms/bagging.html), [boosting](https://dictionaryofml.org/terms/boosting.html), stacking, [base learner](https://dictionaryofml.org/terms/baselearner.html), [random forest](https://dictionaryofml.org/terms/randomforest.html), [variance](https://dictionaryofml.org/terms/variance.html), [correlation](https://dictionaryofml.org/terms/correlation.html).

## References

- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Krogh and Vedelsby (1994). Neural Network Ensembles, Cross Validation, and Active Learning. *Adv. Neural Inf. Process. Syst. (NeurIPS)*.
- Breiman (2001). Random Forests. *Machine Learning*. <https://doi.org/10.1023/A:1010933404324>

## Cite this entry

```bibtex
@misc{dictml_ensemble,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {ensemble},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/ensemble.html}
}
```