Dictionary of Applied Machine Learning: 115 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here
Updated on 2026-11-08
Readiness 100/100
See also independent and identically distributed random variable probability distribution variance $k$-fold cross-validation bagging validation set validation error
The bootstrap is a stochastic algorithm that generates perturbed variants, referred to as bootstrap samples, of a given dataset. Each bootstrap sample is obtained by randomly selecting data points, with replacement, from the original dataset. The bootstrap samples have the same size as the original dataset. However, they might repeat some of the original data points while omitting others. Several bootstrap samples allow the study of the properties of a machine learning (ML) method. For example, a robust ML method delivers similar model parameters for two datasets that differ only in the random fluctuations of the data generation. Bootstrap samples can also be used to estimate a confidence interval for a validation error. Another application of the bootstrap is bagging, where the hypotheses learned from the bootstrap samples are aggregated into one.

bootstrap.py, which recomputes what the entry states: the script block by block · open in ColabB-fiveFive days of weather recordings at Krems an der Donau
are all the data there is. Each day is a data point, its
feature the minimum temperature of the day and its label the
maximum temperature of the same day (Fig. 1).
This dataset is small: it does not allow the training
of a high-dimensional model, and a hypothesis learned
from it depends on which five days happened to be recorded. The bootstrap is a stochastic algorithm, stated
as Algorithm 1, that generates additional
datasets from the recorded one, the so-called bootstrap
samples (Efron and Tibshirani, 1993), and so makes that dependence
measurable. Fig. 1 shows two bootstrap samples
obtained from Algorithm 1. A bootstrap
sample can contain a data point of $\mathcal{D}$ several times,
so it is a list of data points rather than a set. The
bootstrap is based on the independent and identically distributed assumption (i.i.d. assumption).
pythondemos/bootstrap.py
B-empdistThe empirical distribution $\mathbb{P}^{(\mathcal{D})}$ is a fully specified probability distribution and can be sampled as often as wanted. As many data points and as many datasets as needed can therefore be generated from the collected dataset. The gain is copies, not information. Each copy keeps the size $m$ of $\mathcal{D}$, because the quantity being imitated is the spread of a dataset of that size drawn from $\mathbb{P}$. Nor does a copy contain new data points: $\mathbb{P}^{(\mathcal{D})}$ places all its mass on the $m$ data points observed, so no draw can produce one that $\mathcal{D}$ does not already contain. A variant, the smoothed bootstrap, draws from a smoothed version of $\mathbb{P}^{(\mathcal{D})}$ in which each data point is replaced by a small bump-shaped probability density function (pdf); it produces data points that $\mathcal{D}$ does not contain, which the bootstrap itself never does.
Sampling $m$ data points from
$\mathbb{P}^{(\mathcal{D})}$ is the same as drawing $m$ times
from $\mathcal{D}$ with replacement, which is what
Algorithm 1 does: every draw is made from
the whole of $\mathcal{D}$, so a data point can be drawn twice
and another not at all (Fig. 2).
The number $B$ of bootstrap samples is limited only by
the computation they cost.
A bootstrap sample therefore holds about $0.63\,m$ distinct data points and overlaps $\mathcal{D}$ heavily. That overlap is what separates the bootstrap from $k$-fold cross-validation ($k$-fold CV), which partitions $\mathcal{D}$ into blocks that do not overlap at all. Used naively to estimate the prediction error of a learned hypothesis, the bootstrap therefore evaluates it on data points that entered its own training, and the estimate comes out optimistic (Hastie et al., 2009, Sect. 7.11).
Machine learning (ML) training methods can be applied separately to each bootstrap sample, giving the learned hypotheses $\widehat{h}^{(1)}, \,\ldots, \,\widehat{h}^{(B)}$. A quantity computed from $\widetilde{\mathcal{D}}^{(b)}$ is a bootstrap replication (Efron and Tibshirani, 1993), such as the hypothesis $\widehat{h}^{(b)}$ learned by empirical risk minimization (ERM) from $\widetilde{\mathcal{D}}^{(b)}$ or its prediction $\widehat{h}^{(b)}({\mathbf x})$ for a fixed data point. The spread of the $B$ bootstrap replications estimates how much the output of the ML method changes due to random fluctuations in the data generation (Hastie et al., 2009, Sect. 8.2.1).
B-valciFig. 3 shows the bootstrap for
learning a linear hypothesis from weather data recorded at
Krems an der Donau during $2024$, $m = 366$ days. Each
dot is one data point, its feature the minimum
temperature and its label the maximum temperature of the
day; the figure draws $20$ of them. The thick line is the
hypothesis learned from all $366$ by ERM with the
squared error loss. The thin lines show
bootstrap replications $\widehat{h}^{(b)}$,
$b = 1, \,\ldots, \,B$, learned by the
same ERM problem from bootstrap samples. Their spread is
narrow here, and the $95\%$ confidence interval for the slope
is $[1.05, 1.15]$.
pythondemos/bootstrap.py
Bagging is the bootstrap put to a different use: instead of measuring the spread of $\widehat{h}^{(1)}, \,\ldots, \,\widehat{h}^{(B)}$, it averages them into one hypothesis, which is how a random forest is built from decision trees.
See also: empirical distribution, independent and identically distributed, random variable, probability distribution, variance, $k$-fold cross-validation, bagging, confidence interval, validation, validation set, validation error.
@misc{dictml_bootstrap,
author = {Jung, Alexander},
editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
title = {bootstrap},
howpublished = {Dictionary of Applied Machine Learning (course edition)},
year = {2026},
doi = {10.5281/zenodo.21569296},
note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-13},
url = {https://dictionaryofml.org/terms/bootstrap.html}
}