Dictionary of Applied Machine Learning: 107 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

ensemble

Updated on 2026-10-22

Readiness 100/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Cite this entry

See also bagging boosting random forest variance

An ensemble method combines several machine learning (ML) methods, each referred to as a base learner. Ensemble methods differ in how they construct the base learners and in how they combine their results. Bagging trains each base learner on a resampled copy of the training set. Boosting runs them sequentially, each one fitted to the errors of the previous ones. Stacking uses base learners to train different models on the same training set and learns an aggregation rule. For the squared error loss the gain is exact: the error of the average prediction is the average error of the base learners minus their average disagreement, so base learners that disagree are what makes an ensemble worth forming.

Figure produced by the Python demo of the entry ensemble
Figure written by the entry's Python demo ensemble.py, which recomputes what the entry states: the script block by block · open in Colab

Definition

B-combinePredicting the maximum temperature of a day at the weather station Krems an der Donau from the minimum temperature of the same day can be done with a linear model, with a decision tree, or with an artificial neural network (ANN). Fig. 1 shows all three learned from the $366$ days of $2020$. None of them is the best everywhere: the linear hypothesis is too stiff at the cold end, the decision tree answers with a staircase, and the ANN bends where the other two cannot. Model selection would keep one of them and discard the rest. Is there a better way to combine these base learners instead of choosing one of them?

Figure 1 of the entry ensemble
Figure 1: Three base learners on one task: the maximum temperature of a day at Krems an der Donau against the minimum temperature of the same day, for the $366$ days of $2020$. The linear model, the decision tree and the ANN are each fit to the same $244$ training days. On the $122$ held-back days their average squared errors are $26.20$, $27.09$ and $26.13$, so none of them is the best; the average of the three (solid curve) reaches $25.95$. Data generated by pythondemos/ensemble.py
Combining several machine learning (ML) methods into one predictor is an ensemble method: each combined method is a base learner (Hastie et al., 2009, Ch. 16). The base learners can be empirical risk minimization (ERM)-based, using different choices for the loss, model, and training set. Of $\numlearners$ base learners, base learner $\ell$ learns a hypothesis $\learntlocalhypothesis{\ell}$ from its hypothesis space $\localmodel{\ell}$ and its, possibly perturbed, training set $\widetilde{\dataset}^{(\ell)}$, and an aggregation rule $\phi_{\text{agg}}$ combines the outputs of these hypotheses into the final prediction. The aggregation can amount to averaging (in regression), as in Fig. 1, or to a majority vote (in classification); Fig. 2 shows the general shape.
Figure 2 of the entry ensemble
Figure 2: Generic ensemble with three base learners: base learner $\ell \in \{1, 2, 3\}$ uses ERM to learn $\learntlocalhypothesis{\ell} \in \localmodel{\ell}$ from the perturbed training set $\widetilde{\dataset}^{(\ell)}$. A base learner might also use the output of other base learners (dashed arrows). The final hypothesis $\learnthypothesis$ is obtained by aggregating the hypotheses of the base learners via an aggregation rule $\phi_{\text{agg}}$
What the combination buys is exactly the disagreement among the base learners. For the squared error loss, the error of the average prediction is the average error of the base learners minus their average disagreement, where the disagreement of a base learner is the squared difference between its own prediction and the average one (Krogh and Vedelsby, 1994). This is an identity, not an approximation: on the held-back days of Fig. 1 the average member error $26.47$ minus the average disagreement $0.52$ is the $25.95$ that the average prediction actually incurs. Two consequences follow. The average is never worse than its base learners are on average, since a disagreement cannot be negative. And base learners that agree everywhere buy nothing, which is why they are trained on different models or on different data points. Beating the best single base learner is a different and weaker claim: over $50$ repetitions on random halves of the training set, the average did so $22$ times.

B-varThe same point in terms of variance: consider $\numlearners$ base learners whose predictions are identically distributed, each with variance $\sigma^{2}$ and pairwise correlation $\rho$. The average of these predictions has variance (Hastie et al., 2009, Sect. 15.2) \[\rho \sigma^{2} + \frac{1-\rho}{\numlearners}\,\sigma^{2}\text{.}\] The second of these two summands vanishes as $\numlearners$ grows, while the first, set by the correlation between the base learners, limits what averaging can achieve.

Ensemble methods differ in how they construct the base learners and in how they combine their results. Bagging trains each base learner on a random resample of the training set and averages them; it reduces the variance of an unstable base learner such as a deep decision tree, while the systematic part of its error, the bias, is left unchanged (Hastie et al., 2009, Sect. 8.7). A random forest is bagging of decision trees with, in addition, a random choice of the features considered at each split, which lowers the correlation between the trees (Breiman, 2001, Sect. 2.2).

Boosting runs the base learners sequentially, each one fitted to the errors of the previous ones. The number of rounds is a hyperparameter: the training error keeps decreasing while the validation error eventually rises, so boosting overfits when run for too long (Hastie et al., 2009, Sect. 10.12). Stacking trains the base learners on the same training set with different models and learns the aggregation rule itself by a second ML method. The second method is trained on predictions for data points the base learners did not see. Otherwise the base learners that fit the training set best are favored regardless of how they generalize (Hastie et al., 2009, Sect. 8.8).

See also: bagging, boosting, stacking, base learner, random forest, variance, correlation.

References

  1. Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media. doi.org/10.1007/978-0-387-84858-7
  2. Krogh and Vedelsby (1994). Neural Network Ensembles, Cross Validation, and Active Learning. Adv. Neural Inf. Process. Syst. (NeurIPS).
  3. Breiman (2001). Random Forests. Machine Learning. doi.org/10.1023/A:1010933404324

Cite this entry

@misc{dictml_ensemble,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {ensemble},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-10-27},
  url = {https://dictionaryofml.org/terms/ensemble.html}
}