Dictionary of Applied Machine Learning: 107 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

random forest

Updated on 2026-10-22

Readiness 100/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Cite this entry

See also ensemble bagging decision tree bootstrap feature importance variance

A random forest is the instance of bagging whose base learners learn decision trees. Each decision tree is trained on randomly selected data points of the training set, and each base learner also uses only a randomly selected subset of the features. The predictions of the base learners are aggregated by a majority vote in classification or by averaging in regression. The random selection of data points and of features aims at a decorrelation, so that aggregating reduces the variance of the returned prediction. The aggregation of base learners is a form of regularization and counteracts overfitting. A random forest also yields a feature importance score per feature: the fraction of the overall splits that use that feature.

Figure produced by the Python demo of the entry random forest
Figure written by the entry's Python demo randomforest.py, which recomputes what the entry states: the script block by block · open in Colab

Definition

B-treesA bank must decide, for each loan application, whether the applicant is creditworthy. A random forest answers with the majority vote of many different decision trees. A random forest is the instance of bagging whose base learners learn decision trees. As in bagging, each decision tree is trained on randomly selected data points of the training set. Besides the random selection of data points, each base learner also uses only a randomly selected subset of the features of the original data points (Breiman, 2001). The predictions of the base learners are aggregated by a majority vote in classification or by averaging in regression. The scatterplot of Fig. 1 shows a regression example: each data point is one of the 366 days of 2020 at the weather station Krems an der Donau, its feature the minimum and its label the maximum air temperature of the day. Three decision trees, each trained on its own random selection of the days, yield three different piecewise constant hypotheses; the random forest delivers their average.

Figure 1 of the entry randomforest
Figure 1: Scatterplot of the days of 2020 at the weather station Krems an der Donau, with the minimum temperature of a day as feature $\feature$ and its maximum temperature as label $\truelabel$. Three decision trees $\learntlocalhypothesis{1}, \learntlocalhypothesis{2}, \learntlocalhypothesis{3}$, each trained on its own random selection of the days (dotted, dashed and dash-dotted step curves). The thick solid step curve is the random forest $\learnthypothesis$, which delivers the average of the three. Data generated by pythondemos/randomforest.py
This construction works because averaging many high-variance predictions yields one of lower variance, provided the errors of the individual predictions are weakly correlated. A deep decision tree is well suited as a base learner here: it can fit the training set closely (small bias), but its prediction varies strongly with the training set (large variance). Averaging reduces this variance. Consider $\numlearners$ identically distributed predictions, each with variance $\sigma^{2}$ and with pairwise correlation $\rho$. The averaged prediction has variance \[ \rho \sigma^{2} + \frac{1-\rho}{\numlearners}\, \sigma^{2} \text{,} \] which decreases with the number $\numlearners$ of trees toward the lower bound $\rho \sigma^{2}$ set by the correlation (Hastie et al., 2009, Eq. (15.1)). The random selections are drawn independently of each other once the training set is fixed, which is what makes the predictions identically distributed. They are not independent when the underlying dataset is itself a random variable (RV): every decision tree is grown from the same training set, and that shared dependence results in a nonzero correlation $\rho$ in general.

B-tradeThe random selection of data points and the random selection of features both aim at a decorrelation of the predictions that the base learners deliver for a feature vector $\featurevec$ (Fig. 2; Hastie et al., 2009, Sect. 15.2). While withholding features lowers the correlation $\rho$ between the base learners, it raises the variance $\sigma^{2}$ of each of them. The number of features offered to each base learner is therefore worth tuning: too many and the decision trees agree, too few and each of them predicts poorly.

Figure 2 of the entry randomforest
Figure 2: A random forest with three base learners, each learning a decision tree on its own random selection of the data points of the training set. For a feature vector $\featurevec$, the prediction $\learnthypothesis(\featurevec)$ is obtained from an aggregation rule $\phi_{\rm agg}$ applied to the tree predictions $\learntlocalhypothesis{1}(\featurevec), \,\ldots,\, \learntlocalhypothesis{3}(\featurevec)$
The aggregation of base learners is a form of regularization and counteracts overfitting (Hastie et al., 2009, Sect. 15.3.4). A random forest naturally yields a feature importance score for each feature: the fraction of the overall splits that use that feature. Fig. 3 computes these scores for a forest grown on four features of a day at Krems an der Donau.
Figure 3 of the entry randomforest
Figure 3: Feature importance scores of a forest of $60$ decision trees grown on four features of a day at Krems an der Donau, predicting the maximum air temperature. Each base learner was offered randomly selected days and a randomly selected pair of the four features, and every split it made was recorded. The score of a feature is the share of the $420$ splits that test it. Data generated by pythondemos/randomforest.py

See also: ensemble, bagging, decision tree, bootstrap, feature importance, variance, correlation.

References

  1. Breiman (2001). Random Forests. Machine Learning. doi.org/10.1023/A:1010933404324
  2. Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media. doi.org/10.1007/978-0-387-84858-7

Cite this entry

@misc{dictml_randomforest,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {random forest},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-10-27},
  url = {https://dictionaryofml.org/terms/randomforest.html}
}