Dictionary of Applied Machine Learning: 107 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

boosting

Updated on 2026-10-22

Readiness 100/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Cite this entry

See also ensemble bagging

Boosting is an iterative optimization method that sequentially combines base learners, referred to as weak learners. Starting from an initial hypothesis, each iteration adds the hypothesis of a base learner, scaled by a learning rate, to the current hypothesis. This update generalizes the gradient step of gradient-based methods for empirical risk minimization (ERM): each new base learner corrects what the previous ones got wrong. Gradient boosting is the special case of this scheme with decision trees as base learners.

Figure produced by the Python demo of the entry boosting
Figure written by the entry's Python demo boosting.py, which recomputes what the entry states: the script block by block · open in Colab

Definition

A weather station records, for each day, the minimum and the maximum temperature. Predicting the maximum from the minimum is a regression task, and a single decision tree of depth one answers it with two numbers: one prediction below a threshold on the minimum temperature and one above it. Such a tree is too coarse to follow five days recorded at Krems an der Donau (Fig. 1). Boosting adds a second tree that corrects the predictions of the first, a third that corrects what the first two leave over, and so on.

B-kremsBoosting combines a sequence of base learners, each trained to correct what the previous ones got wrong, into one accurate hypothesis (referred to as a strong learner) (Drucker, 1997; Hastie et al., 2009, Ch. 10; Ridgeway et al., 1999; Schapire, 1999).

Figure 1 of the entry boosting
Figure 1: Three iterations of boosting with decision trees of depth one on five days recorded at the station Krems an der Donau: the feature is the minimum temperature of the day, the label its maximum. The constant $\widetilde{\hypothesis}^{(0)}$ (dotted line) is the mean label. Each further iteration adds one tree and one step to the staircase, and the average squared prediction error over the five days falls from $93.1$ to $31.0$, $10.9$ and $7.5$. Data generated by pythondemos/boosting.py
Boosting is an instance of an ensemble method, since it combines base learners that may use different models. Other ensemble methods such as bagging average their base learners. Boosting instead accumulates them in a sequential fashion, each one correcting the predictions of those before it.

Boosting methods start from an initial hypothesis $\widetilde{\hypothesis}^{(0)}$, which is iteratively improved by the update \[ \widetilde{\hypothesis}^{(\iteridx)} = \widetilde{\hypothesis}^{(\iteridx-1)}+ \lrate^{(\iteridx)} \learntlocalhypothesis{\iteridx}\text{.}\] Here, $\lrate^{(\iteridx)}$ denotes a learning rate and $\learntlocalhypothesis{\iteridx}$ is provided by the $\iteridx$-th base learner (see Fig. 2).

Figure 2 of the entry boosting
Figure 2: Boosting constructs a sequence of hypothesis maps. Each new hypothesis map $\widetilde{\hypothesis}^{(\iteridx+1)}$ is obtained from a base learner $\learntlocalhypothesis{\iteridx+1}$ that aims to correct the prediction errors of the current hypothesis $\widetilde{\hypothesis}^{(\iteridx)}$
The boosting update is structurally similar to a gradient step for the model parameters. The boosting update, however, is carried out directly on the hypothesis space, while a gradient step is carried out on a parameter space.

B-gradstepThe $\iteridx$-th base learner is trained to compensate the prediction errors incurred by the current hypothesis on the training set, $\truelabel^{(\sampleidx)} - \widetilde{\hypothesis}^{(\iteridx-1)}(\featurevec^{(\sampleidx)})$ for $\sampleidx = 1, \ldots, \samplesize$. For the squared error loss these errors are the negative gradient of the training error with respect to the current predictions (Friedman, 2001). Its output $\learntlocalhypothesis{\iteridx}$ therefore plays the role of a negative gradient.

B-overfitThe update is a fixed-point iteration on the hypothesis space, $\widetilde{\hypothesis}^{(\iteridx)} = T\big(\widetilde{\hypothesis}^{(\iteridx-1)}\big)$ with the operator $T(\hypothesis) \defeq \hypothesis + \lrate \, B(\hypothesis)$, where $B(\hypothesis)$ is the base learner fitted to the prediction errors of $\hypothesis$ on the training set. A hypothesis is a fixed point of $T$ when the base learner fitted to its prediction errors no longer points downhill, so that a step in its direction would not decrease the training error; boosting halts there (Mason et al., 1999, Sect. 2). If the training error is convex in the predictions and each iteration picks the base learner that decreases it most, every accumulation point of the sequence $\widetilde{\hypothesis}^{(\iteridx)}$ attains the smallest training error the base learners can reach (Mason et al., 1999, Thm. 2).

Boosting methods differ in their choice of base learners for computing the generalized gradients $\learntlocalhypothesis{\iteridx}$. For instance, gradient boosting most commonly uses decision trees as base learners (Hastie et al., 2009, Sect. 10.9).

See also: ensemble, adaptive boosting, gradient boosting, base learner, gradient-based method, bagging, early stopping.

References

  1. Drucker (1997). Improving Regressors using Boosting Techniques. Proc. 14th Int. Conf. Mach. Learn..
  2. Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media. doi.org/10.1007/978-0-387-84858-7
  3. Ridgeway et al. (1999). Boosting Methodology for Regression Problems. Proc. 7th Int. Workshop Artif. Intell. Statist.. proceedings.mlr.press/r2/ridgeway99a.html
  4. Schapire (1999). A brief introduction to boosting. Proc. 16th Int. Joint Conf. Artif. Intell.. www.ijcai.org/Proceedings/99-2/Papers/103.pdf
  5. Friedman (2001). Greedy Function Approximation: A Gradient Boosting Machine. Ann. Statist.. doi.org/10.1214/aos/1013203451
  6. Mason et al. (1999). Boosting Algorithms as Gradient Descent. Adv. Neural Inf. Process. Syst. (NeurIPS).

Cite this entry

@misc{dictml_boosting,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {boosting},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-10-27},
  url = {https://dictionaryofml.org/terms/boosting.html}
}