Dictionary of Applied Machine Learning · boosting

boosting — Python demo

Updated on 2026-11-03

Numerical companion to the entry boosting: it recomputes what the entry states and prints one line per check

Runs boosting with the squared error loss on a task with one feature and a real-valued label and checks the entry's claims numerically: each base learner is fit to the residuals, which are the negative gradient of the training error with respect to the current predictions, so each iteration lowers the training error; with enough iterations the training error becomes arbitrarily small while the validation error eventually rises, so the number of iterations is a hyperparameter chosen by early stopping; and accumulating base learners (boosting) differs from averaging them (bagging). It also runs three iterations on five real temperature measurements from the GeoSphere Austria station Krems an der Donau. The entry's opening figure shows the first tree and the sum after adding a second tree. numpy and matplotlib only (stdlib urllib for the download), fixed seed.

Run it without installing anything:
uv run https://dictionaryofml.org/terms/boosting.py
uv downloads this script and the pinned NumPy and Matplotlib it needs, then runs it; the script fetches any input file it uses. To keep the output files, download boosting.py into a folder and run uv run boosting.py there. With NumPy and Matplotlib already installed, python3 boosting.py, from any directory — it writes its output files into the current directory. Fixed seeds, so the printed numbers reproduce exactly. Download boosting.py · Notebook · Open in Colab

Before running a downloaded copy, scan it: uvx bandit -q -t B102,B307,B602,B605 boosting.py (B102,B307,B602,B605 are the exec, eval and shell checks of bandit) reports nothing: the script runs no shell command and evaluates no generated code.

The script, block by block

One cell per block of the script: the code, and what that code printed when it last ran here

setup

"""
boosting.py — numerical companion to the glossary entry 'boosting'.

Purpose
-------
Runs boosting with the squared error loss on a task with one feature
and a real-valued label and checks the entry's claims numerically: each base learner is fit
to the residuals, which are the negative gradient of the training error
with respect to the current predictions, so each iteration lowers the
training error; with enough iterations the training error becomes
arbitrarily small while the validation error eventually rises, so the
number of iterations is a hyperparameter chosen by early stopping; and
accumulating base learners (boosting) differs from averaging them
(bagging).  It also runs three iterations on five real temperature
measurements from the GeoSphere Austria station Krems an der Donau.
The entry's opening figure shows the first tree and the sum after adding
a second tree.  numpy and matplotlib only
(stdlib urllib for the download), fixed seed.

Setup
-----
Training pairs: m = 30 pairs (x, y) with feature x in [0, 1] and label
y = sin(2 pi x) + 0.5 x + noise; a validation set of 30 further pairs
of the same kind.  Base learners: decision trees of depth one
(one split of the feature range, one constant prediction on each side).
Learning rate eta = 0.5, up to T = 300 iterations, initialization
h^(0) = mean label of the training pairs.

Blocks
------
[B-krems]    Five days of 2024 at the station Krems an der Donau (the
             15th of January, March, May, July and October):
             feature x = the minimum temperature of the day, label
             y = the maximum temperature of the same day.  Two iterations
             of boosting with decision trees of depth one lower the training
             error.  The opening figure shows h^(1) and h^(2).
[B-gradstep] The base learner of iteration t is fit to the residuals
             y - h^(t-1)(x), i.e. to the negative gradient of the training
             error with respect to the predictions; adding it with
             learning rate eta lowers the training error in every
             iteration.
[B-overfit]  The training error falls below 0.02 by T = 300, while the
             validation error reaches its minimum at an earlier iteration
             and is at least 0.03 larger at T = 300 than at that minimum:
             early stopping picks the number of iterations.
[B-bagging]  Averaging 300 base learners, each trained on a random draw
             with replacement from the training pairs (bagging), leaves the
             training error more than five times that of accumulating
             them (boosting).

Outputs
-------
boosting_krems.csv  : x,y of the five Krems days.
boosting_krems_fit.csv : x grid with h^(1) and h^(2) of the two boosting
                      iterations on those five days.
boosting_errors.csv : t, trainerr, valerr of boosting, and the training
                      error of bagging with t base learners.
boosting_fit.csv    : x grid with the label function, the boosted
                      hypothesis after early stopping and after T = 300.
boosting.png        : matplotlib preview of the two panels (checking
                      only).
"""

# /// script
# requires-python = ">=3.10"
# dependencies = ["matplotlib==3.10.7", "numpy==2.3.4"]
# ///

import json
import urllib.request

import numpy as np
import matplotlib

matplotlib.use("Agg")
import matplotlib.pyplot as plt

from pathlib import Path

OUT_DIR = Path(__file__).parent

report = []


def check(name, ok):
    report.append((name, bool(ok)))
    print(f"  [{'ok' if ok else 'FAIL'}] {name}")


rng = np.random.default_rng(0)

m = 30
x_tr = np.sort(rng.uniform(0.0, 1.0, m))
x_va = np.sort(rng.uniform(0.0, 1.0, m))


def truth(xs):
    return np.sin(2.0 * np.pi * xs) + 0.5 * xs


y_tr = truth(x_tr) + 0.3 * rng.standard_normal(m)
y_va = truth(x_va) + 0.3 * rng.standard_normal(m)


def fit_stump(xs, target):
    """Depth-one decision tree: the split and the two constants that
    minimize the squared error of the fit to `target`."""
    order = np.argsort(xs); xs_s, t_s = xs[order], target[order]
    best = (np.inf, None, None, None)
    for i in range(1, len(xs_s)):
        if xs_s[i - 1] == xs_s[i]:
            continue                      # no split separates two equal features
        left, right = t_s[:i].mean(), t_s[i:].mean()
        err = ((t_s[:i] - left) ** 2).sum() + ((t_s[i:] - right) ** 2).sum()
        if err < best[0]:
            best = (err, 0.5 * (xs_s[i - 1] + xs_s[i]), left, right)
    _, split, left, right = best
    return lambda q: np.where(q < split, left, right)


def trainerr(pred, y):
    return float(np.mean((y - pred) ** 2))
    Krems, five days: tmin [-6.4, 2.4, 7.0, 20.8, 6.2], tmax [3.2, 17.5, 24.2, 32.3, 15.8]

B-krems

Five days of 2024 at the station Krems an der Donau (the 15th of January, March, May, July and October): feature x = the minimum temperature of the day, label y = the maximum temperature of the same day. Two iterations of boosting with decision trees of depth one lower the training error. The opening figure shows h^(1) and h^(2).

# Five days of 2024 at Krems an der Donau (GeoSphere Austria, station 3805):
# the minimum temperature of a day as the feature, its maximum as the label.
KREMS_DAYS = ["2024-01-15", "2024-03-15", "2024-05-15", "2024-07-15",
              "2024-10-15"]
KREMS_URL = ("https://dataset.api.hub.geosphere.at/v1/station/historical/"
             "klima-v2-1d?parameters=tlmin,tlmax&station_ids=3805"
             "&start=2024-01-01&end=2024-12-31")
with urllib.request.urlopen(KREMS_URL, timeout=120) as resp:
    payload = json.load(resp)
_cols = payload["features"][0]["properties"]["parameters"]
_stamps = [t[:10] for t in payload["timestamps"]]
_pick = [_stamps.index(d) for d in KREMS_DAYS]
x_kr = np.array([_cols["tlmin"]["data"][i] for i in _pick], dtype=float)
y_kr = np.array([_cols["tlmax"]["data"][i] for i in _pick], dtype=float)
print(f"    Krems, five days: tmin {np.round(x_kr, 1).tolist()}, "
      f"tmax {np.round(y_kr, 1).tolist()}")
check("[B-krems]    five days with both temperatures read",
      len(x_kr) == 5 and not np.isnan(x_kr).any() and not np.isnan(y_kr).any())

ETA_KR, T_KR = 1.0, 2
h_kr = np.full(len(x_kr), y_kr.mean())
kr_learners, kr_err = [], [trainerr(h_kr, y_kr)]
for _ in range(T_KR):
    g = fit_stump(x_kr, y_kr - h_kr)      # fit to the current errors
    kr_learners.append(g)
    h_kr = h_kr + ETA_KR * g(x_kr)
    kr_err.append(trainerr(h_kr, y_kr))
print("    training error over the two iterations: "
      + " -> ".join(f"{e:.2f}" for e in kr_err))
check(f"[B-krems]    every iteration lowers the training error "
      f"({kr_err[0]:.1f} -> {kr_err[-1]:.2f})",
      all(b < a for a, b in zip(kr_err, kr_err[1:])))

with open(OUT_DIR / "boosting_krems.csv", "w") as fh:
    fh.write("x,y\n")
    for a, b in zip(x_kr, y_kr):
        fh.write(f"{a:g},{b:g}\n")
xg_kr = np.linspace(x_kr.min() - 1.5, x_kr.max() + 1.5, 400)


def boosted_kr(xs, n):
    out = np.full(len(xs), y_kr.mean())
    for g in kr_learners[:n]:
        out = out + ETA_KR * g(xs)
    return out


with open(OUT_DIR / "boosting_krems_fit.csv", "w") as fh:
    fh.write("x,h1,h2\n")
    rows = [boosted_kr(xg_kr, n) for n in (1, 2)]
    for i, a in enumerate(xg_kr):
        fh.write(f"{a:.4f}," + ",".join(f"{r[i]:.4f}" for r in rows) + "\n")
  [ok] [B-krems]    five days with both temperatures read
    training error over the two iterations: 93.05 -> 30.97 -> 10.87
  [ok] [B-krems]    every iteration lowers the training error (93.1 -> 10.87)

B-gradstep

The base learner of iteration t is fit to the residuals y - h^(t-1)(x), i.e. to the negative gradient of the training error with respect to the predictions; adding it with learning rate eta lowers the training error in every iteration.

eta, T = 0.5, 300
h_tr = np.full(m, y_tr.mean()); h_va = np.full(m, y_tr.mean())
learners = []
err_tr, err_va = [trainerr(h_tr, y_tr)], [trainerr(h_va, y_va)]
monotone = True
for t in range(1, T + 1):
    residual = y_tr - h_tr                 # negative gradient (up to 2/m)
    g = fit_stump(x_tr, residual)
    learners.append(g)
    h_tr = h_tr + eta * g(x_tr); h_va = h_va + eta * g(x_va)
    err_tr.append(trainerr(h_tr, y_tr)); err_va.append(trainerr(h_va, y_va))
    monotone &= err_tr[-1] < err_tr[-2]
grad_check = np.allclose(residual, -0.5 * m * (-(2.0 / m) * residual))
check(f"[B-gradstep] residuals are the negative gradient of the training "
      f"error (up to the factor 2/m), and all {T} iterations lower it: "
      f"{err_tr[0]:.3f} -> {err_tr[-1]:.4f}", monotone and grad_check)
  [ok] [B-gradstep] residuals are the negative gradient of the training error (up to the factor 2/m), and all 300 iterations lower it: 0.445 -> 0.0101

B-overfit

The training error falls below 0.02 by T = 300, while the validation error reaches its minimum at an earlier iteration and is at least 0.03 larger at T = 300 than at that minimum: early stopping picks the number of iterations.

t_best = int(np.argmin(err_va))
check(f"[B-overfit]  training error {err_tr[-1]:.4f} at T={T}; validation "
      f"error minimal at t={t_best} ({err_va[t_best]:.3f}), larger at T="
      f"{T} ({err_va[-1]:.3f})",
      err_tr[-1] < 0.02 and t_best < T and err_va[-1] > err_va[t_best] + 0.03)
  [ok] [B-overfit]  training error 0.0101 at T=300; validation error minimal at t=22 (0.105), larger at T=300 (0.158)

B-bagging

Averaging 300 base learners, each trained on a random draw with replacement from the training pairs (bagging), leaves the training error more than five times that of accumulating them (boosting).

bag_err = []
bag_sum = np.zeros(m)
for b in range(1, T + 1):
    idx = rng.integers(0, m, m)
    g = fit_stump(x_tr[idx], y_tr[idx])
    bag_sum += g(x_tr)
    bag_err.append(trainerr(bag_sum / b, y_tr))
check(f"[B-bagging]  training error of {T} averaged base learners "
      f"{bag_err[-1]:.3f} vs {err_tr[-1]:.4f} for {T} accumulated ones",
      bag_err[-1] > 5 * err_tr[-1])

# ---------------------------------------------------------------- CSV
with open(OUT_DIR / "boosting_errors.csv", "w") as fh:
    fh.write("t,trainerr,valerr,bagging_trainerr\n")
    for t in range(T + 1):
        be = bag_err[t - 1] if t > 0 else trainerr(np.full(m, y_tr.mean()), y_tr)
        fh.write(f"{t},{err_tr[t]:.4f},{err_va[t]:.4f},{be:.4f}\n")
xg = np.linspace(0.0, 1.0, 201)


def boosted(xs, n):
    out = np.full(len(xs), y_tr.mean())
    for g in learners[:n]:
        out = out + eta * g(xs)
    return out


with open(OUT_DIR / "boosting_fit.csv", "w") as fh:
    fh.write("x,truth,h_early,h_final\n")
    for a, b, c, d in zip(xg, truth(xg), boosted(xg, t_best), boosted(xg, T)):
        fh.write(f"{a:.4f},{b:.4f},{c:.4f},{d:.4f}\n")

# -------------------------------------------------------------- preview
fig, (ax0, ax1, ax2) = plt.subplots(1, 3, figsize=(14.0, 3.9))
ax0.plot(x_kr, y_kr, "ko", ms=6, label="the five days")
for n, ls in ((1, "--"), (2, "-")):
    ax0.plot(xg_kr, boosted_kr(xg_kr, n), "k", ls=ls, lw=1.3,
             label=f"$h^{{({n})}}$")
ax0.set_xlabel("minimum temperature of the day in deg C")
ax0.set_ylabel("maximum temperature of the day in deg C")
ax0.set_title("Krems: adding a second tree")
ax0.legend(frameon=False, fontsize=8, loc="upper left")
ts = np.arange(T + 1)
ax1.plot(ts, err_tr, "k-", lw=1.3, label="training error (boosting)")
ax1.plot(ts, err_va, "k--", lw=1.3, label="validation error (boosting)")
ax1.plot(ts, [err_tr[0]] + bag_err, "k:", lw=1.5, label="training error (bagging)")
ax1.axvline(t_best, color="0.5", lw=0.8, ls="-.", label=f"early stopping, t={t_best}")
ax1.set_xlabel("iteration $t$"); ax1.set_ylabel("average squared error")
ax1.set_title("accumulated vs averaged base learners")
ax1.legend(frameon=False, fontsize=8)
ax2.plot(x_tr, y_tr, "ko", ms=3, label="training pairs")
ax2.plot(xg, truth(xg), "k-", lw=1.0, label="label function")
ax2.plot(xg, boosted(xg, t_best), "k--", lw=1.3, label=f"boosted hypothesis, t={t_best}")
ax2.plot(xg, boosted(xg, T), "k:", lw=1.3, label=f"boosted hypothesis, t={T}")
ax2.set_xlabel("feature $x$"); ax2.set_ylabel("label $y$")
ax2.set_title("boosting overfits when run too long")
ax2.legend(frameon=False, fontsize=8)
fig.tight_layout()
fig.savefig(OUT_DIR / "boosting.png", dpi=110)

n_ok = sum(ok for _, ok in report)
print(f"\n{n_ok}/{len(report)} checks pass")
print(f"wrote boosting_krems.csv, boosting_krems_fit.csv, "
      f"boosting_errors.csv, boosting_fit.csv, boosting.png in {OUT_DIR}")
if n_ok != len(report):
    raise SystemExit(1)
  [ok] [B-bagging]  training error of 300 averaged base learners 0.267 vs 0.0101 for 300 accumulated ones

5/5 checks pass
wrote boosting_krems.csv, boosting_krems_fit.csv, boosting_errors.csv, boosting_fit.csv, boosting.png in /Users/junga1/dictionaryappliedml/pythondemos
Preview figure produced by boosting.py
The preview figure the block B-bagging writes when the script runs