Dictionary of Applied Machine Learning · ensemble

ensemble — Python demo

Updated on 2026-11-17

Numerical companion to the entry ensemble: it recomputes what the entry states and prints one line per check

Three different ML methods are fit to the same task: predict the maximum temperature of a day at the GeoSphere Austria station Krems an der Donau from the minimum temperature of that same day, over the 366 days of 2020. The base learners are a linear model, a decision tree of depth three, and a small artificial neural network. None of the three is best everywhere, which is the entry's opening question: instead of selecting one of them, combine them.

Run it without installing anything:
uv run https://dictionaryofml.org/terms/ensemble.py
uv downloads this script and the pinned NumPy and Matplotlib it needs, then runs it; the script fetches any input file it uses. To keep the output files, download ensemble.py into a folder and run uv run ensemble.py there. With NumPy and Matplotlib already installed, python3 ensemble.py, from any directory — it writes its output files into the current directory. Fixed seeds, so the printed numbers reproduce exactly. Download ensemble.py · Notebook · Open in Colab

Before running a downloaded copy, scan it: uvx bandit -q -t B102,B307,B602,B605 ensemble.py (B102,B307,B602,B605 are the exec, eval and shell checks of bandit) reports nothing: the script runs no shell command and evaluates no generated code.

The script, block by block

One cell per block of the script: the code, and what that code printed when it last ran here

setup

"""
ensemble.py — numerical companion to the glossary entry 'ensemble'.

Purpose
-------
Three different ML methods are fit to the same task: predict the maximum
temperature of a day at the GeoSphere Austria station Krems an der Donau
from the minimum temperature of that same day, over the 366 days of
2020.  The base learners are a linear model, a decision tree of depth
three, and a small artificial neural network.  None of the three is
best everywhere, which is the entry's opening question: instead of
selecting one of them, combine them.

The demo then checks what combining buys, exactly.  For the squared
error loss the error of the average prediction is the average error of
the members MINUS their average disagreement -- an algebraic identity,
verified here to machine precision.  Disagreement is therefore the
whole of the gain, which is what makes "the errors point in different
directions" a checkable statement rather than a figure of speech.  The
weaker claim is also measured, and it does not always hold: the average
beats the mean of its members always, but it beats the BEST member only
sometimes.

Finally, the variance of an average of B identically distributed
predictions with pairwise correlation rho is rho sigma^2 +
(1 - rho) sigma^2 / B: the second term vanishes as B grows, the first
is what correlation leaves behind.  numpy and matplotlib only (stdlib
urllib for the download), fixed seeds.

Blocks
------
[B-days]     The 366 days of 2020 at Krems: feature x = minimum
             temperature of the day, label y = maximum temperature of
             the same day.  Two thirds of the days train, one third is
             held back.  Written to ensemble_days.csv.
[B-learners] Three base learners on the training days: least squares,
             a depth-three regression tree, and a 1-8-1 tanh network
             trained by gradient descent.  Each is fit; none has the
             smallest held-back error everywhere, which is why choosing
             one is a choice.  Their curves go to ensemble_fits.csv.
[B-combine]  The ambiguity decomposition: on the held-back days the
             error of the average equals the average member error minus
             the average disagreement, to machine precision.  The
             average therefore never does worse than its members do on
             average.  Over 50 repetitions with different random
             training halves it beats the BEST member only part of the
             time, which is reported rather than claimed away.
[B-var]      For sigma^2 = 1 and rho = 0.3, the empirical variance of
             the average of B predictions, over 20000 draws, matches
             rho sigma^2 + (1 - rho) sigma^2 / B within 5 percent for
             every B in {1, 2, 3, 5, 10, 20, 50}.
[B-limit]    At B = 50 the variance is within 10 percent of rho sigma^2:
             the correlated part is what averaging cannot remove.
[B-preview]  The matplotlib preview of the three fits and of the
             variance curve.

Outputs
-------
ensemble_days.csv     : x,y,split — the 366 days and which set each is in.
ensemble_days_plot.csv : x,y of every 19th day, the 20 dots of Fig. 1.
ensemble_fits.csv     : x grid with the linear, tree, network and average
                        predictions, for the entry's opening figure.
ensemble_variance.csv : B, empirical variance of the average, formula value.
ensemble.png          : matplotlib preview (checking only).
"""

# /// script
# requires-python = ">=3.10"
# dependencies = ["matplotlib==3.10.7", "numpy==2.3.4"]
# ///

import json
import urllib.request
from pathlib import Path

import numpy as np
import matplotlib

matplotlib.use("Agg")
import matplotlib.pyplot as plt

OUT_DIR = Path(__file__).parent

report = []


def check(name, ok):
    report.append((name, bool(ok)))
    print(f"  [{'ok' if ok else 'FAIL'}] {name}")


def mse(pred, truth):
    return float(np.mean((pred - truth) ** 2))

B-days

The 366 days of 2020 at Krems: feature x = minimum temperature of the day, label y = maximum temperature of the same day. Two thirds of the days train, one third is held back. Written to ensemble_days.csv.

URL = ("https://dataset.api.hub.geosphere.at/v1/station/historical/"
       "klima-v2-1d?parameters=tlmin,tlmax&station_ids=3805"
       "&start=2020-01-01&end=2020-12-31")
with urllib.request.urlopen(URL, timeout=120) as resp:
    payload = json.load(resp)
cols = payload["features"][0]["properties"]["parameters"]
x = np.array(cols["tlmin"]["data"], dtype=float)
y = np.array(cols["tlmax"]["data"], dtype=float)
check("[B-days]     366 days of 2020 downloaded, none missing",
      len(x) == 366 and not np.isnan(x).any() and not np.isnan(y).any())

rng = np.random.default_rng(1)
order = rng.permutation(len(x))
tr, va = order[:244], order[244:]
x_tr, y_tr, x_va, y_va = x[tr], y[tr], x[va], y[va]
print(f"    {len(tr)} training days, {len(va)} held back; "
      f"min temp {x.min():.1f} to {x.max():.1f} deg C")
with open(OUT_DIR / "ensemble_days.csv", "w") as fh:
    fh.write("x,y,split\n")
    for i in range(len(x)):
        fh.write(f"{x[i]:g},{y[i]:g},{'train' if i in set(tr) else 'val'}\n")
# Fig. 1 of the entry draws every 19th day, 20 of the 366, so the dots do
# not hide the four curves; the base learners are learned from the 244
# training days and scored on the 122 held back
PLOT_EVERY = 19
with open(OUT_DIR / "ensemble_days_plot.csv", "w") as fh:
    fh.write("x,y\n")
    for i in range(0, len(x), PLOT_EVERY):
        fh.write(f"{x[i]:g},{y[i]:g}\n")
check("[B-days]     the plotted subset holds 20 days", len(range(0, len(x), PLOT_EVERY)) == 20)
  [ok] [B-days]     366 days of 2020 downloaded, none missing
    244 training days, 122 held back; min temp -7.9 to 19.5 deg C
  [ok] [B-days]     the plotted subset holds 20 days
    linear model    training  25.42   held back  26.20
    decision tree   training  21.79   held back  27.09
    neural network  training  23.94   held back  26.13

B-learners

Three base learners on the training days: least squares, a depth-three regression tree, and a 1-8-1 tanh network trained by gradient descent. Each is fit; none has the smallest held-back error everywhere, which is why choosing one is a choice. Their curves go to ensemble_fits.csv.

def fit_linear(xs, ys):
    w = np.linalg.lstsq(np.c_[xs, np.ones(len(xs))], ys, rcond=None)[0]
    return lambda q: w[0] * np.asarray(q, dtype=float) + w[1]


def fit_tree(xs, ys, depth=3, leaf=8):
    """Regression tree on the single feature: the split minimizing the
    squared error, recursively, down to `depth` levels."""
    if depth == 0 or len(xs) < 2 * leaf or xs.min() == xs.max():
        m = float(ys.mean())
        return lambda q: np.full(np.shape(q), m, dtype=float)
    o = np.argsort(xs)
    xs_s, ys_s = xs[o], ys[o]
    best_err, split = np.inf, None
    for i in range(leaf, len(xs_s) - leaf):
        if xs_s[i - 1] == xs_s[i]:
            continue                      # no split separates equal features
        err = (((ys_s[:i] - ys_s[:i].mean()) ** 2).sum()
               + ((ys_s[i:] - ys_s[i:].mean()) ** 2).sum())
        if err < best_err:
            best_err, split = err, 0.5 * (xs_s[i - 1] + xs_s[i])
    if split is None:
        m = float(ys.mean())
        return lambda q: np.full(np.shape(q), m, dtype=float)
    left = fit_tree(xs[xs < split], ys[xs < split], depth - 1, leaf)
    right = fit_tree(xs[xs >= split], ys[xs >= split], depth - 1, leaf)
    return lambda q: np.where(np.asarray(q, dtype=float) < split,
                              left(q), right(q))


def fit_net(xs, ys, seed=2, hidden=8, steps=4000, lrate=0.5):
    """1 - hidden - 1 network with a tanh layer, trained by gradient
    descent on the standardized data."""
    g = np.random.default_rng(seed)
    mx, sx, my, sy = xs.mean(), xs.std(), ys.mean(), ys.std()
    z, t = (xs - mx) / sx, (ys - my) / sy
    W1, b1 = g.normal(0, 1.0, (1, hidden)), np.zeros(hidden)
    W2, b2 = g.normal(0, 1.0, (hidden, 1)), np.zeros(1)
    for _ in range(steps):
        h = np.tanh(z[:, None] @ W1 + b1)
        out = (h @ W2 + b2).ravel()
        d = (out - t) / len(z)
        gW2, gb2 = h.T @ d[:, None], d.sum(keepdims=True)
        dh = (d[:, None] @ W2.T) * (1 - h ** 2)
        W1 -= lrate * (z[:, None].T @ dh)
        b1 -= lrate * dh.sum(axis=0)
        W2 -= lrate * gW2
        b2 -= lrate * gb2

    def predict(q):
        zq = (np.asarray(q, dtype=float) - mx) / sx
        h = np.tanh(zq[:, None] @ W1 + b1)
        return ((h @ W2 + b2).ravel()) * sy + my
    return predict


learners = [("linear model", fit_linear(x_tr, y_tr)),
            ("decision tree", fit_tree(x_tr, y_tr)),
            ("neural network", fit_net(x_tr, y_tr))]
err_va = np.array([mse(h(x_va), y_va) for _, h in learners])
for (nm, h), e in zip(learners, err_va):
    print(f"    {nm:15s} training {mse(h(x_tr), y_tr):6.2f}   held back {e:6.2f}")
check("[B-learners] all three base learners are fit and differ on the "
      "held-back days",
      len({round(e, 6) for e in err_va}) == 3)

grid = np.linspace(x.min() - 1.0, x.max() + 1.0, 400)
P_grid = np.stack([h(grid) for _, h in learners])
with open(OUT_DIR / "ensemble_fits.csv", "w") as fh:
    fh.write("x,linear,tree,net,average\n")
    for i, q in enumerate(grid):
        fh.write(f"{q:.4f},{P_grid[0][i]:.4f},{P_grid[1][i]:.4f},"
                 f"{P_grid[2][i]:.4f},{P_grid[:, i].mean():.4f}\n")
  [ok] [B-learners] all three base learners are fit and differ on the held-back days
    average member error 26.4740 - disagreement 0.5245 = 25.9495; error of the average 25.9495

B-combine

The ambiguity decomposition: on the held-back days the error of the average equals the average member error minus the average disagreement, to machine precision. The average therefore never does worse than its members do on average. Over 50 repetitions with different random training halves it beats the BEST member only part of the time, which is reported rather than claimed away.

P = np.stack([h(x_va) for _, h in learners])
avg = P.mean(axis=0)
err_avg = mse(avg, y_va)
ambiguity = float(np.mean([np.mean((p - avg) ** 2) for p in P]))
gap = abs((err_va.mean() - ambiguity) - err_avg)
print(f"    average member error {err_va.mean():.4f} - disagreement "
      f"{ambiguity:.4f} = {err_va.mean() - ambiguity:.4f}; "
      f"error of the average {err_avg:.4f}")
check("[B-combine]  error of the average = average member error - average "
      f"disagreement, exactly (gap {gap:.2e})", gap < 1e-9)
check("[B-combine]  the average is therefore no worse than its members "
      "on average", err_avg <= err_va.mean() + 1e-12)

reps, beats_best = 50, 0
for r in range(reps):
    g = np.random.default_rng(100 + r)
    hs = []
    for build in (fit_linear, fit_tree, fit_net):
        s = g.choice(len(x_tr), len(x_tr) // 2, replace=False)
        hs.append(build(x_tr[s], y_tr[s]))
    Q = np.stack([h(x_va) for h in hs])
    e = [mse(q, y_va) for q in Q]
    beats_best += mse(Q.mean(axis=0), y_va) < min(e)
print(f"    over {reps} repetitions on random training halves, the average "
      f"beat the BEST of its three members {beats_best} times")
check("[B-combine]  beating the best single member is not guaranteed, and is "
      f"reported as measured ({beats_best}/{reps})", 0 <= beats_best <= reps)
  [ok] [B-combine]  error of the average = average member error - average disagreement, exactly (gap 3.55e-15)
  [ok] [B-combine]  the average is therefore no worse than its members on average
    over 50 repetitions on random training halves, the average beat the BEST of its three members 22 times
  [ok] [B-combine]  beating the best single member is not guaranteed, and is reported as measured (22/50)

B-var

For sigma^2 = 1 and rho = 0.3, the empirical variance of the average of B predictions, over 20000 draws, matches rho sigma^2 + (1 - rho) sigma^2 / B within 5 percent for every B in {1, 2, 3, 5, 10, 20, 50}.

sigma2, rho, draws = 1.0, 0.3, 20000
Bs = [1, 2, 3, 5, 10, 20, 50]
emp, formula = [], []
vrng = np.random.default_rng(0)
for B in Bs:
    shared = vrng.standard_normal((draws, 1)) * np.sqrt(rho * sigma2)
    own = vrng.standard_normal((draws, B)) * np.sqrt((1 - rho) * sigma2)
    emp.append(float((shared + own).mean(axis=1).var()))
    formula.append(rho * sigma2 + (1 - rho) * sigma2 / B)
dev = max(abs(e - f) / f for e, f in zip(emp, formula))
check(f"[B-var]      variance of the average matches rho s^2 + (1-rho) s^2/B; "
      f"largest relative deviation {dev:.3f}", dev < 0.05)
  [ok] [B-var]      variance of the average matches rho s^2 + (1-rho) s^2/B; largest relative deviation 0.031

B-limit

At B = 50 the variance is within 10 percent of rho sigma^2: the correlated part is what averaging cannot remove.

check(f"[B-limit]    at B=50 the variance is {emp[-1]:.3f}, leaving the "
      f"correlated part rho s^2 = {rho * sigma2:.1f}",
      abs(emp[-1] - rho * sigma2) < 0.1 * rho * sigma2)

with open(OUT_DIR / "ensemble_variance.csv", "w") as fh:
    fh.write("B,empirical,formula\n")
    for B, e, f in zip(Bs, emp, formula):
        fh.write(f"{B},{e:.4f},{f:.4f}\n")
  [ok] [B-limit]    at B=50 the variance is 0.315, leaving the correlated part rho s^2 = 0.3

8/8 checks pass
wrote ensemble_days.csv, ensemble_fits.csv, ensemble_variance.csv, ensemble.png in /Users/junga1/dictionaryappliedml/pythondemos

B-preview

The matplotlib preview of the three fits and of the variance curve.

fig, (ax, ax2) = plt.subplots(1, 2, figsize=(10.2, 4.0))
ax.plot(x_tr, y_tr, "o", color="0.75", ms=3, label="training days")
styles = [("-.", "linear model"), (":", "decision tree"), ("--", "neural network")]
for (ls, nm), row in zip(styles, P_grid):
    ax.plot(grid, row, "k", ls=ls, lw=1.3, label=nm)
ax.plot(grid, P_grid.mean(axis=0), "k-", lw=2.0, label="average of the three")
ax.set_xlabel("minimum temperature of the day in deg C")
ax.set_ylabel("maximum temperature of the day in deg C")
ax.set_title("Krems 2020: three base learners, one task")
ax.legend(frameon=False, fontsize=8, loc="upper left")
ax2.plot(Bs, formula, "k-", lw=1.3,
         label=r"$\rho\sigma^2 + (1-\rho)\sigma^2/B$")
ax2.plot(Bs, emp, "ko", mfc="none", ms=6, label="empirical variance")
ax2.axhline(rho * sigma2, color="k", ls=":", lw=1,
            label=r"$\rho\sigma^2$, the shared part")
ax2.set_xscale("log")
ax2.set_xlabel("number of base learners B")
ax2.set_ylabel("variance of the average prediction")
ax2.set_title(r"averaging keeps the correlated part ($\rho = 0.3$)")
ax2.legend(frameon=False, fontsize=8)
fig.tight_layout()
fig.savefig(OUT_DIR / "ensemble.png", dpi=110)

n_ok = sum(ok for _, ok in report)
print(f"\n{n_ok}/{len(report)} checks pass")
print(f"wrote ensemble_days.csv, ensemble_fits.csv, ensemble_variance.csv, "
      f"ensemble.png in {OUT_DIR}")
if n_ok != len(report):
    raise SystemExit(1)
Preview figure produced by ensemble.py
The preview figure the block B-preview writes when the script runs