Dictionary of Applied Machine Learning · bootstrap

bootstrap — Python demo

Updated on 2026-11-03

Numerical companion to the entry bootstrap: it recomputes what the entry states and prints one line per check

The 366 days of 2024 at Krems an der Donau, each a (tmin, tmax) pair, stand in for the dataset D; five days of 2022 carry the entry's opening figure. The blocks below read the days as realizations of i.i.d. RVs, replace their unknown probability distribution by the empirical distribution of D, and draw from it: Algorithm 1 of the entry. Self-contained (numpy/matplotlib only, data fetched from the GeoSphere Austria archive), fixed seed.

Run it without installing anything:
uv run https://dictionaryofml.org/terms/bootstrap.py
uv downloads this script and the pinned NumPy and Matplotlib it needs, then runs it; the script fetches any input file it uses. To keep the output files, download bootstrap.py into a folder and run uv run bootstrap.py there. With NumPy and Matplotlib already installed, python3 bootstrap.py, from any directory — it writes its output files into the current directory. Fixed seeds, so the printed numbers reproduce exactly. Download bootstrap.py · Notebook · Open in Colab

Before running a downloaded copy, scan it: uvx bandit -q -t B102,B307,B602,B605 bootstrap.py (B102,B307,B602,B605 are the exec, eval and shell checks of bandit) reports nothing: the script runs no shell command and evaluates no generated code.

The script, block by block

One cell per block of the script: the code, and what that code printed when it last ran here

setup

"""
bootstrap.py — numerical companion to the glossary entry 'bootstrap'.

The 366 days of 2024 at Krems an der Donau, each a (tmin, tmax) pair,
stand in for the dataset D; five days of 2022 carry the entry's opening
figure. The blocks below read the days as realizations of i.i.d. RVs,
replace their unknown probability distribution by the empirical
distribution of D, and draw from it: Algorithm 1 of the entry.
Self-contained (numpy/matplotlib only, data fetched from the GeoSphere
Austria archive), fixed seed.

Blocks
------
[B-data]    366 days of (tmin, tmax) at Krems, 2024. This cloud of
            points is the dataset D; the entry's picture of it is the
            scatterplot of its Fig. 3.
[B-five]    Five days of 2022, the 15th of February, April, June,
            August and October, carry the entry's opening figure: the
            five (tmin, tmax) pairs on top, and below them two bootstrap
            samples drawn from them with replacement, each of five days,
            one day drawn twice and another omitted. Drawn with its own
            generator so the later blocks keep their stream.
[B-empdist] The empirical distribution P^(D) puts mass 1/m on each of
            the m days. Drawing m days from it is drawing from D with
            replacement, and a draw holds about 0.63 m distinct days.
            P^(D) is a complete distribution, so it can be sampled as
            often as wanted: drawing 100 m days from it is no problem,
            and the mean of such a draw converges to the mean of D, not
            to the mean of the unknown P. The gain is replicates, not
            information.
[B-fit]     ERM with the squared error loss learns tmax = w0 + w1 tmin
            on D, which is least squares. Learning again on each of
            B = 500 bootstrap samples gives B slopes whose 2.5th to
            97.5th percentile range is a confidence interval for the
            slope; it widens like 1/sqrt(m) when m is cut to a quarter.
[B-valci]   A threshold rule predicts a frost night (tmin < 0) from the
            day's tmax. The rule is learned once on a training set and
            its validation error measured on a validation set;
            resampling that validation set B times, with the rule held
            fixed, gives B estimates of the risk, whose sample mean and
            sample variance (and whose percentiles) give a confidence
            interval around the single validation error.
[B-smooth]  Replacing each day by a small Gaussian bump turns the m
            spikes of P^(D) into a density estimate (preview only).
            Sampling that density instead of the spikes is the smoothed
            bootstrap: it produces days that D does not contain, which
            the plain bootstrap never does.

Outputs
-------
bootstrap_krems.csv    : tmin, tmax of the 366 days (figure data).
bootstrap_krems_plot.csv : every 19th of those days, the 20 dots of Fig. 3.
bootstrap_fivedays.csv : tmin, tmax and index 1-5 of the five days.
bootstrap_star1.csv    : the five draws of the first bootstrap sample,
                         plotted positions (a repeated draw is stacked
                         FIVE_DY degrees above the first).
bootstrap_star2.csv    : the same for the second bootstrap sample.
bootstrap_fit.csv      : the learned line as one two-point segment.
bootstrap_lines.csv    : 25 bootstrap lines, two-point segments separated
                         by empty lines.
bootstrap.png          : preview figure (checking only).

Data generated by pythondemos/bootstrap.py.
"""

# /// script
# requires-python = ">=3.10"
# dependencies = ["matplotlib==3.10.7", "numpy==2.3.4"]
# ///

import json
import urllib.request
import numpy as np
import matplotlib

matplotlib.use("Agg")
import matplotlib.pyplot as plt
from pathlib import Path

OUT_DIR = Path(__file__).parent
ARCHIVE = "https://dataset.api.hub.geosphere.at/v1/station/historical/"
STATION = 3805                      # Krems an der Donau
NR_BOOTSTRAP = 500
SEED = 20220101
# The opening figure shows each bootstrap sample in its own panel over the
# five days, so two markers would land on one spot where a day was drawn
# twice: a repeated draw is stacked FIVE_DY degrees above the first, a
# drawing device the caption states (FIVE_DX = 0: no sideways shift).
FIVE_DATES = ["2022-02-15", "2022-04-15", "2022-06-15", "2022-08-15",
              "2022-10-15"]
FIVE_DX, FIVE_DY = 0.0, 1.8
# The figure is meant to show the two things a draw with replacement does, so
# FIVE_OFFSET was picked as the first generator offset whose two draws each
# repeat exactly one day and omit exactly one, omit different days, and cover
# all five between them. The checks below hold the picture to that.
FIVE_OFFSET = 2

rng = np.random.default_rng(SEED)
report = []


def check(name, ok):
    report.append((name, bool(ok)))
    print(f"  [{'ok' if ok else 'FAIL'}] {name}")


def fetch(parameters, start, end):
    """Daily parameters of one station from the GeoSphere Austria archive."""
    url = (f"{ARCHIVE}klima-v2-1d?parameters={','.join(parameters)}"
           f"&station_ids={STATION}&start={start}&end={end}")
    with urllib.request.urlopen(url, timeout=180) as resp:
        payload = json.load(resp)
    values = payload["features"][0]["properties"]["parameters"]
    return np.stack([np.array(values[p]["data"], dtype=float)
                     for p in parameters], axis=1)


def least_squares_line(x, y):
    """Slope and intercept of the least-squares fit of y on x."""
    design = np.stack([np.ones_like(x), x], axis=1)
    weights = np.linalg.lstsq(design, y, rcond=None)[0]
    return weights[1], weights[0]


def write_segments(name, header, segments):
    """Polylines, one blank line between them (pgfplots 'empty line=jump')."""
    with open(OUT_DIR / name, "w") as f:
        f.write(header + "\n")
        for k, seg in enumerate(segments):
            if k:
                f.write("\n")
            for px, py in seg:
                f.write(f"{px:.4f},{py:.4f}\n")

B-data

366 days of (tmin, tmax) at Krems, 2024. This cloud of points is the dataset D; the entry's picture of it is the scatterplot of its Fig. 3.

days = fetch(["tlmin", "tlmax"], "2024-01-01", "2024-12-31")
tmin, tmax = days[:, 0], days[:, 1]
m = tmin.size
print(f"[B-data] {m} days at Krems an der Donau, 2024; "
      f"tmin {tmin.min():.1f} to {tmin.max():.1f}, "
      f"tmax {tmax.min():.1f} to {tmax.max():.1f} degrees")
check("[B-data] one full (leap) year, no gaps", m == 366 and not np.isnan(days).any())
with open(OUT_DIR / "bootstrap_krems.csv", "w") as f:
    f.write("tmin,tmax\n")
    for a, b in zip(tmin, tmax):
        f.write(f"{a:.1f},{b:.1f}\n")
# the entry's Fig. 3 draws every 19th day, 20 of the 366, so the dots do
# not hide the lines; every number of the entry is computed on all 366
PLOT_EVERY = 19
with open(OUT_DIR / "bootstrap_krems_plot.csv", "w") as f:
    f.write("tmin,tmax\n")
    for a, b in zip(tmin[::PLOT_EVERY], tmax[::PLOT_EVERY]):
        f.write(f"{a:.1f},{b:.1f}\n")
check("[B-data] the plotted subset holds 20 days", tmin[::PLOT_EVERY].size == 20)
[B-data] 366 days at Krems an der Donau, 2024; tmin -10.1 to 20.8, tmax -3.6 to 34.3 degrees
  [ok] [B-data] one full (leap) year, no gaps
  [ok] [B-data] the plotted subset holds 20 days

B-five

Five days of 2022, the 15th of February, April, June, August and October, carry the entry's opening figure: the five (tmin, tmax) pairs on top, and below them two bootstrap samples drawn from them with replacement, each of five days, one day drawn twice and another omitted. Drawn with its own generator so the later blocks keep their stream.

days22 = fetch(["tlmin", "tlmax"], "2022-01-01", "2022-12-31")
five = np.array([(np.datetime64(d) - np.datetime64("2022-01-01")).astype(int)
                 for d in FIVE_DATES])
check("[B-five] the five dates fall inside 2022",
      five.min() >= 0 and five.max() < days22.shape[0])
five_tmin, five_tmax = days22[five, 0], days22[five, 1]
with open(OUT_DIR / "bootstrap_fivedays.csv", "w") as f:
    f.write("tmin,tmax,idx\n")
    for k in range(five.size):
        f.write(f"{five_tmin[k]:.1f},{five_tmax[k]:.1f},{k + 1}\n")
# own generator: inserting draws into the shared stream would move every
# number the later blocks commit, the slope interval of the entry included
rng_five = np.random.default_rng(SEED + FIVE_OFFSET)
omitted = []
for sample, (name, side) in enumerate(
        [("bootstrap_star1.csv", -1.0), ("bootstrap_star2.csv", 1.0)]):
    drawn = rng_five.integers(0, five.size, size=five.size)
    seen = np.zeros(five.size, dtype=int)
    with open(OUT_DIR / name, "w") as f:
        f.write("tmin,tmax,idx\n")
        for i in drawn:
            f.write(f"{five_tmin[i] + side * FIVE_DX:.2f},"
                    f"{five_tmax[i] + seen[i] * FIVE_DY:.2f},{i + 1}\n")
            seen[i] += 1
    omitted.append({int(i) + 1 for i in range(five.size) if i not in drawn})
    print(f"[B-five] bootstrap sample {sample + 1} draws days "
          f"{sorted(int(i) + 1 for i in drawn)}, "
          f"omits {sorted(omitted[-1])}")
    check(f"[B-five] bootstrap sample {sample + 1} holds five days, "
          "repeats exactly one and omits exactly one",
          drawn.size == five.size and np.unique(drawn).size == five.size - 1)
check("[B-five] the two bootstrap samples omit different days and cover "
      "all five between them",
      omitted[0] != omitted[1] and not (omitted[0] & omitted[1]))
  [ok] [B-five] the five dates fall inside 2022
[B-five] bootstrap sample 1 draws days [2, 3, 3, 4, 5], omits [1]
  [ok] [B-five] bootstrap sample 1 holds five days, repeats exactly one and omits exactly one
[B-five] bootstrap sample 2 draws days [1, 2, 2, 4, 5], omits [3]
  [ok] [B-five] bootstrap sample 2 holds five days, repeats exactly one and omits exactly one
  [ok] [B-five] the two bootstrap samples omit different days and cover all five between them

B-empdist

The empirical distribution P^(D) puts mass 1/m on each of the m days. Drawing m days from it is drawing from D with replacement, and a draw holds about 0.63 m distinct days. P^(D) is a complete distribution, so it can be sampled as often as wanted: drawing 100 m days from it is no problem, and the mean of such a draw converges to the mean of D, not to the mean of the unknown P. The gain is replicates, not information.

print("[B-empdist] P^(D) puts mass 1/m on each day; drawing from it is "
      "drawing from D with replacement")
idx = rng.integers(0, m, size=m)
check("[B-empdist] a draw of m days holds about 0.63 m distinct days",
      abs(np.unique(idx).size / m - (1 - np.exp(-1.0))) < 0.05)
check("[B-empdist] every drawn day is a day of D",
      np.all(np.isin(tmin[idx], tmin)))
huge = rng.integers(0, m, size=100 * m)
check("[B-empdist] a draw of 100 m days is no problem, and its mean "
      "converges to the mean of D, not to anything new",
      abs(tmax[huge].mean() - tmax.mean()) < 0.1)
check("[B-empdist] such a draw still contains no day outside D",
      np.unique(tmax[huge]).size <= np.unique(tmax).size)
[B-empdist] P^(D) puts mass 1/m on each day; drawing from it is drawing from D with replacement
  [ok] [B-empdist] a draw of m days holds about 0.63 m distinct days
  [ok] [B-empdist] every drawn day is a day of D
  [ok] [B-empdist] a draw of 100 m days is no problem, and its mean converges to the mean of D, not to anything new
  [ok] [B-empdist] such a draw still contains no day outside D

B-fit

ERM with the squared error loss learns tmax = w0 + w1 tmin on D, which is least squares. Learning again on each of B = 500 bootstrap samples gives B slopes whose 2.5th to 97.5th percentile range is a confidence interval for the slope; it widens like 1/sqrt(m) when m is cut to a quarter.

slope, intercept = least_squares_line(tmin, tmax)
print(f"[B-fit] tmax = {intercept:.2f} + {slope:.2f} tmin on all {m} days")
slopes = np.empty(NR_BOOTSTRAP)
lines = []
for b in range(NR_BOOTSTRAP):
    draw = rng.integers(0, m, size=m)
    slopes[b], icept = least_squares_line(tmin[draw], tmax[draw])
    if b < 25:
        lines.append([(tmin.min(), icept + slopes[b] * tmin.min()),
                      (tmin.max(), icept + slopes[b] * tmin.max())])
lo, hi = np.percentile(slopes, [2.5, 97.5])
print(f"[B-fit] 95% confidence interval for the slope: [{lo:.3f}, {hi:.3f}]")
check("[B-fit] the interval covers the slope learned from all of D",
      lo <= slope <= hi)
check("[B-fit] the interval is narrow, a small fraction of the slope",
      (hi - lo) / slope < 0.3)
quarter = rng.choice(m, size=m // 4, replace=False)
narrow = np.array([least_squares_line(*(lambda d: (tmin[quarter][d],
                                                   tmax[quarter][d]))(
    rng.integers(0, m // 4, size=m // 4)))[0] for _ in range(NR_BOOTSTRAP)])
wide = np.percentile(narrow, 97.5) - np.percentile(narrow, 2.5)
print(f"[B-fit] on a quarter of the days the interval is {wide / (hi - lo):.1f} "
      "times as wide")
check("[B-fit] cutting m to a quarter roughly doubles the interval",
      1.4 < wide / (hi - lo) < 2.8)
# the learned line goes in its own file: the book figure draws it thick and
# the replicates thin, so one file with one style would hide the difference
write_segments("bootstrap_fit.csv", "tmin,tmax",
               [[(tmin.min(), intercept + slope * tmin.min()),
                 (tmin.max(), intercept + slope * tmin.max())]])
write_segments("bootstrap_lines.csv", "tmin,tmax", lines)
[B-fit] tmax = 10.02 + 1.10 tmin on all 366 days
[B-fit] 95% confidence interval for the slope: [1.047, 1.150]
  [ok] [B-fit] the interval covers the slope learned from all of D
  [ok] [B-fit] the interval is narrow, a small fraction of the slope
[B-fit] on a quarter of the days the interval is 2.1 times as wide
  [ok] [B-fit] cutting m to a quarter roughly doubles the interval

B-valci

A threshold rule predicts a frost night (tmin < 0) from the day's tmax. The rule is learned once on a training set and its validation error measured on a validation set; resampling that validation set B times, with the rule held fixed, gives B estimates of the risk, whose sample mean and sample variance (and whose percentiles) give a confidence interval around the single validation error.

frost = (tmin < 0.0).astype(int)
order = rng.permutation(m)
train, val = order[:250], order[250:]
grid = np.linspace(tmax.min(), tmax.max(), 400)
errors = [(np.mean((tmax[train] <= t).astype(int) != frost[train]), t)
          for t in grid]
threshold = min(errors)[1]
predict = lambda x: (x <= threshold).astype(int)
err_val = float(np.mean(predict(tmax[val]) != frost[val]))
print(f"[B-valci] threshold {threshold:.1f} degrees learned on "
      f"{train.size} days; validation error on the {val.size} validation "
      f"days {err_val:.3f}")
check("[B-valci] the learned rule is better than always predicting "
      "no frost", err_val < frost[val].mean())
errs = np.array([float(np.mean(predict(tmax[val][d]) != frost[val][d]))
                 for d in rng.integers(0, val.size,
                                       size=(NR_BOOTSTRAP, val.size))])
err_lo, err_hi = np.percentile(errs, [2.5, 97.5])
err_mean, err_sd = errs.mean(), errs.std(ddof=1)
print(f"[B-valci] {NR_BOOTSTRAP} bootstrap estimates of the risk: sample mean "
      f"{err_mean:.3f}, sample standard deviation {err_sd:.3f}; 95% interval "
      f"[{err_lo:.3f}, {err_hi:.3f}] by percentiles, "
      f"[{err_mean - 1.96 * err_sd:.3f}, {err_mean + 1.96 * err_sd:.3f}] "
      "by mean and standard deviation")
check("[B-valci] the percentile interval covers the validation error",
      err_lo <= err_val <= err_hi)
check("[B-valci] the sample mean of the estimates is the validation error "
      "up to a small fraction of the interval",
      abs(err_mean - err_val) < 0.2 * (err_hi - err_lo))
check("[B-valci] mean plus/minus 1.96 standard deviations and the "
      "percentiles give nearly the same interval",
      abs(err_mean - 1.96 * err_sd - err_lo) < 0.02
      and abs(err_mean + 1.96 * err_sd - err_hi) < 0.02)
check("[B-valci] the interval is several points wide, which the single "
      "number does not show", 0.02 < err_hi - err_lo < 0.2)
check("[B-valci] the hypothesis never changes: only the validation set is "
      "redrawn", predict(np.array([threshold])).item() == 1)
[B-valci] threshold 6.8 degrees learned on 250 days; validation error on the 116 validation days 0.129
  [ok] [B-valci] the learned rule is better than always predicting no frost
[B-valci] 500 bootstrap estimates of the risk: sample mean 0.130, sample standard deviation 0.032; 95% interval [0.069, 0.198] by percentiles, [0.067, 0.193] by mean and standard deviation
  [ok] [B-valci] the percentile interval covers the validation error
  [ok] [B-valci] the sample mean of the estimates is the validation error up to a small fraction of the interval
  [ok] [B-valci] mean plus/minus 1.96 standard deviations and the percentiles give nearly the same interval
  [ok] [B-valci] the interval is several points wide, which the single number does not show
  [ok] [B-valci] the hypothesis never changes: only the validation set is redrawn

B-smooth

Replacing each day by a small Gaussian bump turns the m spikes of P^(D) into a density estimate (preview only). Sampling that density instead of the spikes is the smoothed bootstrap: it produces days that D does not contain, which the plain bootstrap never does.

h_min, h_max = 1.4, 1.8                       # kernel widths, degrees
gx = np.linspace(tmin.min() - 3, tmin.max() + 3, 160)
gy = np.linspace(tmax.min() - 3, tmax.max() + 3, 160)
GX, GY = np.meshgrid(gx, gy)
dens = np.zeros_like(GX)
for a, b in zip(tmin, tmax):
    dens += np.exp(-0.5 * (((GX - a) / h_min) ** 2 + ((GY - b) / h_max) ** 2))
dens /= m * 2 * np.pi * h_min * h_max
cell = (gx[1] - gx[0]) * (gy[1] - gy[0])
print(f"[B-smooth] kernel widths {h_min} and {h_max} degrees; the smeared "
      f"density integrates to {dens.sum() * cell:.3f}")
check("[B-smooth] the density integrates to one", abs(dens.sum() * cell - 1) < 0.02)
counts, edges = np.histogram(tmin, bins=20)
mode_tmin = 0.5 * (edges[counts.argmax()] + edges[counts.argmax() + 1])
peak_tmin = gx[np.unravel_index(dens.argmax(), dens.shape)[1]]
check(f"[B-smooth] its peak ({peak_tmin:.1f} degrees) sits where the days "
      f"pile up ({mode_tmin:.1f}), not at the median ({np.median(tmin):.1f})",
      abs(peak_tmin - mode_tmin) < 1.5)
smoothed = (tmin[rng.integers(0, m, size=20000)]
            + h_min * rng.standard_normal(20000))
check("[B-smooth] the smoothed bootstrap produces days outside D, which "
      "the plain bootstrap never does",
      not np.any(np.isin(smoothed, tmin)))
levels = np.array([0.1, 0.4, 0.7]) * dens.max()

# ------------------------------------------------------------- preview
fig, ax = plt.subplots(1, 4, figsize=(17.0, 3.6))
ax[3].plot(five_tmin, five_tmax, "o", mfc="none", color="black", ms=7,
           label="the five days")
for k, (a, b) in enumerate(zip(five_tmin, five_tmax), start=1):
    ax[3].annotate(str(k), (a, b), textcoords="offset points",
                   xytext=(6, 5), fontsize=8)
for name, mk, lab in [("bootstrap_star1.csv", "s", "bootstrap sample 1"),
                      ("bootstrap_star2.csv", "^", "bootstrap sample 2")]:
    pts = np.loadtxt(OUT_DIR / name, delimiter=",", skiprows=1)
    ax[3].plot(pts[:, 0], pts[:, 1], mk, mfc="none", color="black", ms=5,
               label=lab)
ax[3].set_xlabel("minimum temperature (degC)")
ax[3].set_ylabel("maximum temperature (degC)")
ax[3].set_title("[B-five] five days, and two draws from them", fontsize=10)
ax[3].legend(frameon=False, fontsize=8, loc="upper left")
ax[0].plot(tmin, tmax, ".", color="0.55", markersize=2.5, label="day")
for seg in lines[:25]:
    ax[0].plot([seg[0][0], seg[1][0]], [seg[0][1], seg[1][1]],
               "-", color="0.3", linewidth=0.4)
ax[0].plot([tmin.min(), tmin.max()],
           [intercept + slope * tmin.min(), intercept + slope * tmin.max()],
           "k-", linewidth=1.8, label="learned line")
ax[0].plot([], [], "-", color="0.3", linewidth=0.4, label="bootstrap lines")
ax[0].set_xlabel("minimum temperature (degC)")
ax[0].set_ylabel("maximum temperature (degC)")
ax[0].set_title("[B-fit] the dataset and the spread of the fits", fontsize=10)
ax[0].legend(frameon=False, fontsize=8, loc="upper left")
ax[1].plot(tmin, tmax, ".", color="0.75", markersize=2)
contours = ax[1].contour(GX, GY, dens, levels=levels, colors="black",
                         linewidths=0.9)
ax[1].set_xlabel("minimum temperature (degC)")
ax[1].set_ylabel("maximum temperature (degC)")
ax[1].set_title("[B-smooth] the same days, smeared into a density",
                fontsize=10)
ax[2].hist(slopes, bins=30, color="white", edgecolor="black")
ax[2].axvline(slope, color="black", linestyle="-", linewidth=1.5,
              label="slope on all days")
ax[2].axvline(lo, color="black", linestyle="--", linewidth=1.0,
              label="95% interval")
ax[2].axvline(hi, color="black", linestyle="--", linewidth=1.0)
ax[2].set_xlabel("slope of the bootstrap line")
ax[2].set_ylabel("number of bootstrap samples")
ax[2].set_title("[B-fit] the slopes of 500 replicates", fontsize=10)
ax[2].legend(frameon=False, fontsize=8)
fig.tight_layout()
fig.savefig(OUT_DIR / "bootstrap.png", dpi=110)
print(f"\n{sum(ok for _, ok in report)}/{len(report)} checks passed")
assert all(ok for _, ok in report)
[B-smooth] kernel widths 1.4 and 1.8 degrees; the smeared density integrates to 0.999
  [ok] [B-smooth] the density integrates to one
  [ok] [B-smooth] its peak (2.4 degrees) sits where the days pile up (3.0), not at the median (7.2)
  [ok] [B-smooth] the smoothed bootstrap produces days outside D, which the plain bootstrap never does

22/22 checks passed
Preview figure produced by bootstrap.py
The preview figure the block B-smooth writes when the script runs