Dictionary of Applied Machine Learning

random forest

Updated on 2026-09-07

Typeset PDF version — the authoritative form of this entry

Python demo — a script that recomputes what this entry states and prints one line per check

A random forest is an ensemble of decision trees. Each tree is trained on a bootstrap sample of the training set, and each split may choose only among a small random subset of the features. The predictions of the trees are aggregated by a majority vote in classification or by averaging in regression. The two sources of randomness decorrelate the trees, so that averaging reduces the variance of the aggregated prediction. Adding trees does not lead to overfitting: the generalization error converges to a limit that is bounded via the strength of the trees and the correlation between them. A random forest also yields a widely used form of feature importance, read off from the features its trees split on.

Definition

B-treesA bank must decide, for each loan application, whether the applicant is creditworthy. A random forest answers by asking many different decision trees and following their majority: it is an ensemble of decision trees, each trained on a bootstrap sample of the training set (as in bootstrap aggregating (bagging)) and further randomized by allowing each split to choose only among a small random subset of the features (Breiman, 2001). The predictions of the trees are aggregated by a majority vote in classification or by averaging in regression. Fig. 1 shows a regression example: each data point is one day at the weather station Krems, its feature the minimum and its label the maximum air temperature of the day. Three decision trees, each trained on its own bootstrap sample of the days, yield three different piecewise constant hypotheses; the random forest averages them.

Figure 1 of the entry randomforest
Figure 1: Scatterplot of the days of 2024 at the weather station Krems, with the minimum temperature of a day as feature and its maximum temperature as label. Three decision trees $\learntlocalhypothesis{1}, \learntlocalhypothesis{2}, \learntlocalhypothesis{3}$, each trained on its own bootstrap sample of the days, are piecewise constant maps; the random forest $\learnthypothesis = \frac{1}{3}\big(\learntlocalhypothesis{1} + \learntlocalhypothesis{2} + \learntlocalhypothesis{3}\big)$ averages them. Data generated by pythondemos/randomforest.py
The ensemble idea behind this construction is that averaging many individually unreliable predictions yields a more reliable one, provided the errors of the individual predictions do not align. A deep decision tree is an ideal base learner for this purpose: it can fit the training set closely (small bias), but its prediction varies strongly with the training set (large variance). Averaging attacks precisely this variance: for $B$ identically distributed predictions, each with variance $\sigma^{2}$ and with pairwise correlation $\rho$, the averaged prediction has variance \[ \rho \sigma^{2} + \frac{1-\rho}{B}\, \sigma^{2} \text{,} \] which decreases with the number $B$ of trees toward the floor $\rho \sigma^{2}$ set by the correlation (Hastie et al., 2009, Sect. 15.2). Averaging leaves the bias unchanged, since each tree has the same expected prediction. In Fig. 1, the averaged curve is visibly less ragged than each single tree, and its squared-error risk on the depicted days is smaller than the average of the trees' risks.

The two sources of randomness serve one purpose: they make the trees less alike (Fig. 2). Averaging reduces the variance of a prediction most when the averaged predictions are weakly correlated, and restricting each split to a random subset of features prevents all trees from reusing the same dominant features (Hastie et al., 2009, Sect. 15.2). A random forest also reports which features its trees split on most productively — a widely used form of feature importance.

Figure 2 of the entry randomforest
Figure 2: A random forest with three decision trees. Each tree is trained on its own bootstrap sample of the training set, with each split restricted to a random subset of the features. For a feature vector $\featurevec$, the prediction $\learnthypothesis(\featurevec)$ is the majority vote of the tree predictions $\learntlocalhypothesis{1}(\featurevec), \,\ldots,\, \learntlocalhypothesis{3}(\featurevec)$
A random forest also admits precise guarantees. As trees are added, the generalization error of the forest converges almost surely to a limiting value: a random forest does not suffer from overfitting as the number of trees grows (Breiman, 2001, Thm. 1.2). The limiting error is at most $\bar{\rho}\,(1-s^{2})/s^{2}$, where the strength $s$ is the expected vote margin (the expected gap between the fraction of trees that vote for the correct label of a data point and the largest fraction that votes for any wrong one) and $\bar{\rho}$ is the mean correlation between the raw margin functions of two independently drawn trees (Breiman, 2001, Thm. 2.3). The bound is loose, but it names the two levers that the randomization operates on: strong trees and weak correlation.

The averaging also yields stability: the learned hypothesis depends only weakly on any single data point of the training set. Consider the subbagged version of any machine learning (ML) method with predictions in $[0,1]$: each base learner is trained on a share $p$ of the $\samplesize$ data points, drawn without replacement, and the predictions are averaged over all such subsamples. Then, removing a single data point from the training set changes the prediction at a test point by more than $\varepsilon$ for at most a $\delta$-fraction of the removed data points, for every pair $(\varepsilon, \delta)$ with $\delta \varepsilon^{2} \geq \frac{1}{4(\samplesize-1)} \cdot \frac{p}{1-p}$ (Soloff et al., 2024, Thm. 8). This guarantee holds for every training set and requires nothing of the base learner beyond the bounded predictions — the stability is created by the averaging itself.

In the credit-scoring application, the majority vote over hundreds of trees is far less sensitive to a few unusual customer records than any single decision tree, and the feature importance readout indicates which applicant features drive the decisions.

See also: ensemble, bootstrap aggregating, decision tree, bootstrap, feature importance, variance, stability.

References

  1. Breiman (2001). Random Forests. Machine Learning. doi.org/10.1023/A:1010933404324
  2. Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media. doi.org/10.1007/978-0-387-84858-7
  3. Soloff et al. (2024). Bagging Provides Assumption-free Stability. J. Mach. Learn. Res.. www.jmlr.org/papers/v25/23-0536.html

Cite this entry

@misc{dictml_randomforest,
  author = {Jung, Alexander and Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {random forest},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-09-11},
  url = {https://dictionaryofml.org/terms/randomforest.html}
}