Dictionary of Applied Machine Learning · ensemble
Updated on 2026-11-17
Numerical companion to the entry ensemble: it recomputes what the entry states and prints one line per check
Three different ML methods are fit to the same task: predict the maximum temperature of a day at the GeoSphere Austria station Krems an der Donau from the minimum temperature of that same day, over the 366 days of 2020. The base learners are a linear model, a decision tree of depth three, and a small artificial neural network. None of the three is best everywhere, which is the entry's opening question: instead of selecting one of them, combine them.
Run it without installing anything:uv run https://dictionaryofml.org/terms/ensemble.py
uv downloads this script and the pinned NumPy and Matplotlib it needs, then runs it; the script fetches any input file it uses. To keep the output files, download ensemble.py into a folder and run uv run ensemble.py there. With NumPy and Matplotlib already installed, python3 ensemble.py, from any directory — it writes its output files into the current directory. Fixed seeds, so the printed numbers reproduce exactly. Download ensemble.py · Notebook · Open in Colab
Before running a downloaded copy, scan it: uvx bandit -q -t B102,B307,B602,B605 ensemble.py (B102,B307,B602,B605 are the exec, eval and shell checks of bandit) reports nothing: the script runs no shell command and evaluates no generated code.
One cell per block of the script: the code, and what that code printed when it last ran here
"""
ensemble.py — numerical companion to the glossary entry 'ensemble'.
Purpose
-------
Three different ML methods are fit to the same task: predict the maximum
temperature of a day at the GeoSphere Austria station Krems an der Donau
from the minimum temperature of that same day, over the 366 days of
2020. The base learners are a linear model, a decision tree of depth
three, and a small artificial neural network. None of the three is
best everywhere, which is the entry's opening question: instead of
selecting one of them, combine them.
The demo then checks what combining buys, exactly. For the squared
error loss the error of the average prediction is the average error of
the members MINUS their average disagreement -- an algebraic identity,
verified here to machine precision. Disagreement is therefore the
whole of the gain, which is what makes "the errors point in different
directions" a checkable statement rather than a figure of speech. The
weaker claim is also measured, and it does not always hold: the average
beats the mean of its members always, but it beats the BEST member only
sometimes.
Finally, the variance of an average of B identically distributed
predictions with pairwise correlation rho is rho sigma^2 +
(1 - rho) sigma^2 / B: the second term vanishes as B grows, the first
is what correlation leaves behind. numpy and matplotlib only (stdlib
urllib for the download), fixed seeds.
Blocks
------
[B-days] The 366 days of 2020 at Krems: feature x = minimum
temperature of the day, label y = maximum temperature of
the same day. Two thirds of the days train, one third is
held back. Written to ensemble_days.csv.
[B-learners] Three base learners on the training days: least squares,
a depth-three regression tree, and a 1-8-1 tanh network
trained by gradient descent. Each is fit; none has the
smallest held-back error everywhere, which is why choosing
one is a choice. Their curves go to ensemble_fits.csv.
[B-combine] The ambiguity decomposition: on the held-back days the
error of the average equals the average member error minus
the average disagreement, to machine precision. The
average therefore never does worse than its members do on
average. Over 50 repetitions with different random
training halves it beats the BEST member only part of the
time, which is reported rather than claimed away.
[B-var] For sigma^2 = 1 and rho = 0.3, the empirical variance of
the average of B predictions, over 20000 draws, matches
rho sigma^2 + (1 - rho) sigma^2 / B within 5 percent for
every B in {1, 2, 3, 5, 10, 20, 50}.
[B-limit] At B = 50 the variance is within 10 percent of rho sigma^2:
the correlated part is what averaging cannot remove.
[B-preview] The matplotlib preview of the three fits and of the
variance curve.
Outputs
-------
ensemble_days.csv : x,y,split — the 366 days and which set each is in.
ensemble_days_plot.csv : x,y of every 19th day, the 20 dots of Fig. 1.
ensemble_fits.csv : x grid with the linear, tree, network and average
predictions, for the entry's opening figure.
ensemble_variance.csv : B, empirical variance of the average, formula value.
ensemble.png : matplotlib preview (checking only).
"""
# /// script
# requires-python = ">=3.10"
# dependencies = ["matplotlib==3.10.7", "numpy==2.3.4"]
# ///
import json
import urllib.request
from pathlib import Path
import numpy as np
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
OUT_DIR = Path(__file__).parent
report = []
def check(name, ok):
report.append((name, bool(ok)))
print(f" [{'ok' if ok else 'FAIL'}] {name}")
def mse(pred, truth):
return float(np.mean((pred - truth) ** 2))
The 366 days of 2020 at Krems: feature x = minimum temperature of the day, label y = maximum temperature of the same day. Two thirds of the days train, one third is held back. Written to ensemble_days.csv.
URL = ("https://dataset.api.hub.geosphere.at/v1/station/historical/"
"klima-v2-1d?parameters=tlmin,tlmax&station_ids=3805"
"&start=2020-01-01&end=2020-12-31")
with urllib.request.urlopen(URL, timeout=120) as resp:
payload = json.load(resp)
cols = payload["features"][0]["properties"]["parameters"]
x = np.array(cols["tlmin"]["data"], dtype=float)
y = np.array(cols["tlmax"]["data"], dtype=float)
check("[B-days] 366 days of 2020 downloaded, none missing",
len(x) == 366 and not np.isnan(x).any() and not np.isnan(y).any())
rng = np.random.default_rng(1)
order = rng.permutation(len(x))
tr, va = order[:244], order[244:]
x_tr, y_tr, x_va, y_va = x[tr], y[tr], x[va], y[va]
print(f" {len(tr)} training days, {len(va)} held back; "
f"min temp {x.min():.1f} to {x.max():.1f} deg C")
with open(OUT_DIR / "ensemble_days.csv", "w") as fh:
fh.write("x,y,split\n")
for i in range(len(x)):
fh.write(f"{x[i]:g},{y[i]:g},{'train' if i in set(tr) else 'val'}\n")
# Fig. 1 of the entry draws every 19th day, 20 of the 366, so the dots do
# not hide the four curves; the base learners are learned from the 244
# training days and scored on the 122 held back
PLOT_EVERY = 19
with open(OUT_DIR / "ensemble_days_plot.csv", "w") as fh:
fh.write("x,y\n")
for i in range(0, len(x), PLOT_EVERY):
fh.write(f"{x[i]:g},{y[i]:g}\n")
check("[B-days] the plotted subset holds 20 days", len(range(0, len(x), PLOT_EVERY)) == 20)
[ok] [B-days] 366 days of 2020 downloaded, none missing
244 training days, 122 held back; min temp -7.9 to 19.5 deg C
[ok] [B-days] the plotted subset holds 20 days
linear model training 25.42 held back 26.20
decision tree training 21.79 held back 27.09
neural network training 23.94 held back 26.13
Three base learners on the training days: least squares, a depth-three regression tree, and a 1-8-1 tanh network trained by gradient descent. Each is fit; none has the smallest held-back error everywhere, which is why choosing one is a choice. Their curves go to ensemble_fits.csv.
def fit_linear(xs, ys):
w = np.linalg.lstsq(np.c_[xs, np.ones(len(xs))], ys, rcond=None)[0]
return lambda q: w[0] * np.asarray(q, dtype=float) + w[1]
def fit_tree(xs, ys, depth=3, leaf=8):
"""Regression tree on the single feature: the split minimizing the
squared error, recursively, down to `depth` levels."""
if depth == 0 or len(xs) < 2 * leaf or xs.min() == xs.max():
m = float(ys.mean())
return lambda q: np.full(np.shape(q), m, dtype=float)
o = np.argsort(xs)
xs_s, ys_s = xs[o], ys[o]
best_err, split = np.inf, None
for i in range(leaf, len(xs_s) - leaf):
if xs_s[i - 1] == xs_s[i]:
continue # no split separates equal features
err = (((ys_s[:i] - ys_s[:i].mean()) ** 2).sum()
+ ((ys_s[i:] - ys_s[i:].mean()) ** 2).sum())
if err < best_err:
best_err, split = err, 0.5 * (xs_s[i - 1] + xs_s[i])
if split is None:
m = float(ys.mean())
return lambda q: np.full(np.shape(q), m, dtype=float)
left = fit_tree(xs[xs < split], ys[xs < split], depth - 1, leaf)
right = fit_tree(xs[xs >= split], ys[xs >= split], depth - 1, leaf)
return lambda q: np.where(np.asarray(q, dtype=float) < split,
left(q), right(q))
def fit_net(xs, ys, seed=2, hidden=8, steps=4000, lrate=0.5):
"""1 - hidden - 1 network with a tanh layer, trained by gradient
descent on the standardized data."""
g = np.random.default_rng(seed)
mx, sx, my, sy = xs.mean(), xs.std(), ys.mean(), ys.std()
z, t = (xs - mx) / sx, (ys - my) / sy
W1, b1 = g.normal(0, 1.0, (1, hidden)), np.zeros(hidden)
W2, b2 = g.normal(0, 1.0, (hidden, 1)), np.zeros(1)
for _ in range(steps):
h = np.tanh(z[:, None] @ W1 + b1)
out = (h @ W2 + b2).ravel()
d = (out - t) / len(z)
gW2, gb2 = h.T @ d[:, None], d.sum(keepdims=True)
dh = (d[:, None] @ W2.T) * (1 - h ** 2)
W1 -= lrate * (z[:, None].T @ dh)
b1 -= lrate * dh.sum(axis=0)
W2 -= lrate * gW2
b2 -= lrate * gb2
def predict(q):
zq = (np.asarray(q, dtype=float) - mx) / sx
h = np.tanh(zq[:, None] @ W1 + b1)
return ((h @ W2 + b2).ravel()) * sy + my
return predict
learners = [("linear model", fit_linear(x_tr, y_tr)),
("decision tree", fit_tree(x_tr, y_tr)),
("neural network", fit_net(x_tr, y_tr))]
err_va = np.array([mse(h(x_va), y_va) for _, h in learners])
for (nm, h), e in zip(learners, err_va):
print(f" {nm:15s} training {mse(h(x_tr), y_tr):6.2f} held back {e:6.2f}")
check("[B-learners] all three base learners are fit and differ on the "
"held-back days",
len({round(e, 6) for e in err_va}) == 3)
grid = np.linspace(x.min() - 1.0, x.max() + 1.0, 400)
P_grid = np.stack([h(grid) for _, h in learners])
with open(OUT_DIR / "ensemble_fits.csv", "w") as fh:
fh.write("x,linear,tree,net,average\n")
for i, q in enumerate(grid):
fh.write(f"{q:.4f},{P_grid[0][i]:.4f},{P_grid[1][i]:.4f},"
f"{P_grid[2][i]:.4f},{P_grid[:, i].mean():.4f}\n")
[ok] [B-learners] all three base learners are fit and differ on the held-back days
average member error 26.4740 - disagreement 0.5245 = 25.9495; error of the average 25.9495
The ambiguity decomposition: on the held-back days the error of the average equals the average member error minus the average disagreement, to machine precision. The average therefore never does worse than its members do on average. Over 50 repetitions with different random training halves it beats the BEST member only part of the time, which is reported rather than claimed away.
P = np.stack([h(x_va) for _, h in learners])
avg = P.mean(axis=0)
err_avg = mse(avg, y_va)
ambiguity = float(np.mean([np.mean((p - avg) ** 2) for p in P]))
gap = abs((err_va.mean() - ambiguity) - err_avg)
print(f" average member error {err_va.mean():.4f} - disagreement "
f"{ambiguity:.4f} = {err_va.mean() - ambiguity:.4f}; "
f"error of the average {err_avg:.4f}")
check("[B-combine] error of the average = average member error - average "
f"disagreement, exactly (gap {gap:.2e})", gap < 1e-9)
check("[B-combine] the average is therefore no worse than its members "
"on average", err_avg <= err_va.mean() + 1e-12)
reps, beats_best = 50, 0
for r in range(reps):
g = np.random.default_rng(100 + r)
hs = []
for build in (fit_linear, fit_tree, fit_net):
s = g.choice(len(x_tr), len(x_tr) // 2, replace=False)
hs.append(build(x_tr[s], y_tr[s]))
Q = np.stack([h(x_va) for h in hs])
e = [mse(q, y_va) for q in Q]
beats_best += mse(Q.mean(axis=0), y_va) < min(e)
print(f" over {reps} repetitions on random training halves, the average "
f"beat the BEST of its three members {beats_best} times")
check("[B-combine] beating the best single member is not guaranteed, and is "
f"reported as measured ({beats_best}/{reps})", 0 <= beats_best <= reps)
[ok] [B-combine] error of the average = average member error - average disagreement, exactly (gap 3.55e-15)
[ok] [B-combine] the average is therefore no worse than its members on average
over 50 repetitions on random training halves, the average beat the BEST of its three members 22 times
[ok] [B-combine] beating the best single member is not guaranteed, and is reported as measured (22/50)
For sigma^2 = 1 and rho = 0.3, the empirical variance of the average of B predictions, over 20000 draws, matches rho sigma^2 + (1 - rho) sigma^2 / B within 5 percent for every B in {1, 2, 3, 5, 10, 20, 50}.
sigma2, rho, draws = 1.0, 0.3, 20000
Bs = [1, 2, 3, 5, 10, 20, 50]
emp, formula = [], []
vrng = np.random.default_rng(0)
for B in Bs:
shared = vrng.standard_normal((draws, 1)) * np.sqrt(rho * sigma2)
own = vrng.standard_normal((draws, B)) * np.sqrt((1 - rho) * sigma2)
emp.append(float((shared + own).mean(axis=1).var()))
formula.append(rho * sigma2 + (1 - rho) * sigma2 / B)
dev = max(abs(e - f) / f for e, f in zip(emp, formula))
check(f"[B-var] variance of the average matches rho s^2 + (1-rho) s^2/B; "
f"largest relative deviation {dev:.3f}", dev < 0.05)
[ok] [B-var] variance of the average matches rho s^2 + (1-rho) s^2/B; largest relative deviation 0.031
At B = 50 the variance is within 10 percent of rho sigma^2: the correlated part is what averaging cannot remove.
check(f"[B-limit] at B=50 the variance is {emp[-1]:.3f}, leaving the "
f"correlated part rho s^2 = {rho * sigma2:.1f}",
abs(emp[-1] - rho * sigma2) < 0.1 * rho * sigma2)
with open(OUT_DIR / "ensemble_variance.csv", "w") as fh:
fh.write("B,empirical,formula\n")
for B, e, f in zip(Bs, emp, formula):
fh.write(f"{B},{e:.4f},{f:.4f}\n")
[ok] [B-limit] at B=50 the variance is 0.315, leaving the correlated part rho s^2 = 0.3 8/8 checks pass wrote ensemble_days.csv, ensemble_fits.csv, ensemble_variance.csv, ensemble.png in /Users/junga1/dictionaryappliedml/pythondemos
The matplotlib preview of the three fits and of the variance curve.
fig, (ax, ax2) = plt.subplots(1, 2, figsize=(10.2, 4.0))
ax.plot(x_tr, y_tr, "o", color="0.75", ms=3, label="training days")
styles = [("-.", "linear model"), (":", "decision tree"), ("--", "neural network")]
for (ls, nm), row in zip(styles, P_grid):
ax.plot(grid, row, "k", ls=ls, lw=1.3, label=nm)
ax.plot(grid, P_grid.mean(axis=0), "k-", lw=2.0, label="average of the three")
ax.set_xlabel("minimum temperature of the day in deg C")
ax.set_ylabel("maximum temperature of the day in deg C")
ax.set_title("Krems 2020: three base learners, one task")
ax.legend(frameon=False, fontsize=8, loc="upper left")
ax2.plot(Bs, formula, "k-", lw=1.3,
label=r"$\rho\sigma^2 + (1-\rho)\sigma^2/B$")
ax2.plot(Bs, emp, "ko", mfc="none", ms=6, label="empirical variance")
ax2.axhline(rho * sigma2, color="k", ls=":", lw=1,
label=r"$\rho\sigma^2$, the shared part")
ax2.set_xscale("log")
ax2.set_xlabel("number of base learners B")
ax2.set_ylabel("variance of the average prediction")
ax2.set_title(r"averaging keeps the correlated part ($\rho = 0.3$)")
ax2.legend(frameon=False, fontsize=8)
fig.tight_layout()
fig.savefig(OUT_DIR / "ensemble.png", dpi=110)
n_ok = sum(ok for _, ok in report)
print(f"\n{n_ok}/{len(report)} checks pass")
print(f"wrote ensemble_days.csv, ensemble_fits.csv, ensemble_variance.csv, "
f"ensemble.png in {OUT_DIR}")
if n_ok != len(report):
raise SystemExit(1)

B-preview writes when the script runs