Dictionary of Applied Machine Learning

ensemble

Updated on 2026-09-06

Typeset PDF version — the authoritative form of this entry

An ensemble method combines several machine learning (ML) methods, each referred to as a base learner, into one predictor. The predictions of the base learners are aggregated by averaging in regression or by a majority vote in classification. The combination often predicts more reliably than any single base learner. Ensemble methods differ in how they construct the base learners: bootstrap aggregating (bagging) trains each on a resampled copy of the training set, boosting runs them sequentially so that each corrects the errors of its predecessors, and stacking trains different models on the same training set.

Definition

For each incoming email, three different spam filters cast a vote, and the message is moved to the spam folder when at least two of them say spam. Such a combination of machine learning (ML) methods is an ensemble method: each combined method is a base learner, and the combination often predicts more reliably than any single base learner (Hastie et al., 2009, Ch. 16). The base learners can be empirical risk minimization (ERM)-based, using different choices for the loss, model, and training set. The aggregation of their predictions can amount to averaging (in regression) or to a majority vote (in classification), as in the spam example (Fig. 1).

Figure 1 of the entry ensemble
Figure 1: Generic ensemble with three base learners: base learner $\ell \in \{1, 2, 3\}$ uses ERM to learn $\learntlocalhypothesis{\ell} \in \hypospace^{(\ell)}$ from the perturbed training set $\widetilde{\dataset}^{(\ell)}$. A base learner might also use the output of other base learners (dashed). The final hypothesis $\learnthypothesis$ is obtained by aggregating the hypotheses of the base learners via an aggregation rule $\phi_{\text{agg}}$
Different ensemble methods use different constructions for the base learners. For example, bootstrap aggregating (bagging) methods (such as a random forest) use random sampling to construct slightly different training sets for each base learner. On the other hand, boosting methods run the base learners sequentially, i.e., each base learner tries to correct the prediction errors of the previous ones. A third family of ensemble methods is stacking, where base learners are trained on the same training set but with different models.

See also: bootstrap aggregating, boosting, stacking, base learner, random forest.

References

  1. Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media. doi.org/10.1007/978-0-387-84858-7

Cite this entry

@misc{dictml_ensemble,
  author = {Jung, Alexander and Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {ensemble},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-09-11},
  url = {https://dictionaryofml.org/terms/ensemble.html}
}