Dictionary of Applied Machine Learning

boosting

Updated on 2026-09-06

Typeset PDF version — the authoritative form of this entry

Boosting is an iterative optimization method that learns an accurate hypothesis by sequentially combining less accurate base learners, referred to as weak learners. Starting from an initialization, each iteration adds the output of a base learner, scaled by a learning rate, to the current hypothesis. This update generalizes the gradient step of gradient-based methods for empirical risk minimization (ERM): the output of the base learner acts as a negative generalized gradient. The fixed points of the update are hypotheses at which no base learner can reduce the training error further. Gradient boosting instantiates this scheme with decision trees as base learners.

Definition

An online advertiser must predict, for each impression, whether the user will click on the shown advertisement. Widely deployed predictors for this task are built by boosting: an iterative optimization method that learns an accurate hypothesis map (or strong learner) by sequentially combining less accurate base learners (referred to as weak learners) (Drucker, 1997; Hastie et al., 2009, Chap. 10; Ridgeway et al., 1999; Schapire, 1999). Boosting can be understood as a generalization of gradient-based methods for empirical risk minimization (ERM) using parametric models and smooth loss functions (Friedman, 2001).

In particular, starting from an initialization $\widetilde{\hypothesis}$, boosting methods construct a sequence of hypotheses $\widetilde{\hypothesis}^{(\iteridx)}$, $\iteridx=1,\,\ldots$, via a generalized gradient step \[ \widetilde{\hypothesis}^{(\iteridx)} = \widetilde{\hypothesis}^{(\iteridx-1)}+ \lrate^{(\iteridx)} \learnthypothesis^{(\iteridx)}\text{.}\] Here, $\lrate^{(\iteridx)}$ denotes a learning rate and $\learnthypothesis^{(\iteridx)}$ is provided by the $\iteridx$th base learner (see Fig. 1). Comparing the above update with the plain gradient step suggests viewing $\learnthypothesis^{(\iteridx)}$ as a (negative) generalized gradient.

Figure 1 of the entry boosting
Figure 1: Boosting methods construct a sequence of hypothesis maps via a generalized gradient step. This generalized gradient step uses the output of base learners
The update is a fixed-point iteration on the hypothesis space: its fixed points are hypotheses at which the base learners provide no further descent direction, so the sequence stalls exactly where no base learner can reduce the training error further. Boosting methods differ in their choice of base learners for computing the generalized gradients $\learnthypothesis^{(\iteridx)}$. For instance, gradient boosting — behind many deployed click-through predictors — most commonly uses decision trees as base learners.

See also: ensemble, adaptive boosting, gradient boosting, base learner, gradient-based method.

References

  1. Drucker (1997). Improving Regressors using Boosting Techniques. Proc. 14th Int. Conf. Mach. Learn..
  2. Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media. doi.org/10.1007/978-0-387-84858-7
  3. Ridgeway et al. (1999). Boosting Methodology for Regression Problems. Proc. 7th Int. Workshop Artif. Intell. Statist.. proceedings.mlr.press/r2/ridgeway99a.html
  4. Schapire (1999). A brief introduction to boosting. Proc. 16th Int. Joint Conf. Artif. Intell.. www.ijcai.org/Proceedings/99-2/Papers/103.pdf
  5. Friedman (2001). Greedy Function Approximation: A Gradient Boosting Machine. Ann. Statist.. doi.org/10.1214/aos/1013203451

Cite this entry

@misc{dictml_boosting,
  author = {Jung, Alexander and Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {boosting},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-09-11},
  url = {https://dictionaryofml.org/terms/boosting.html}
}