B-fig writes when the script runsDictionary of Applied Machine Learning · base learner
Numerical companion to the entry base learner: it recomputes what the entry states and prints one line per check
A base learner reads a training set and returns a learned hypothesis. This demo makes that map explicit and then uses learned hypotheses in the three ensemble constructions named in the entry: bagging trains regression stumps on bootstrap samples and averages them; boosting trains regression stumps sequentially on residual labels and adds them; stacking uses out-of-fold predictions of three base learners as features for a linear aggregation rule.
Run it without installing anything:uv run https://dictionaryofml.org/terms/baselearner.py
uv downloads this script and the pinned NumPy and Matplotlib it needs, then runs it; the script fetches any input file it uses. To keep the output files, download baselearner.py into a folder and run uv run baselearner.py there. With NumPy and Matplotlib already installed, python3 baselearner.py, from any directory — it writes its output files into the current directory. Fixed seeds, so the printed numbers reproduce exactly. Download baselearner.py · Notebook · Open in Colab
One cell per block of the script: the code, and what that code printed when it last ran here
"""Numerical companion to the glossary entry 'base learner'.
Purpose
-------
A base learner reads a training set and returns a learned hypothesis. This
demo makes that map explicit and then uses learned hypotheses in the three
ensemble constructions named in the entry: bagging trains regression stumps
on bootstrap samples and averages them; boosting trains regression stumps
sequentially on residual labels and adds them; stacking uses out-of-fold
predictions of three base learners as features for a linear aggregation rule.
Self-contained (NumPy + Matplotlib only), with a fixed random seed.
Blocks
------
[B-data] Generate one nonlinear regression dataset and a dense test grid.
[B-base] Define a regression-stump base learner: training data go in and a
piecewise-constant learned hypothesis comes out.
[B-bag] Train stumps independently on bootstrap samples and average them.
[B-boost] Train stumps sequentially to predict the current residual labels.
[B-stack] Build out-of-fold prediction features and learn their aggregation.
[B-fig] Plot the three ensembles and write baselearner.png.
Outputs
-------
baselearner.png : bagging, boosting, and stacking on the same training set
"""
from pathlib import Path
import matplotlib
import numpy as np
matplotlib.use("Agg")
import matplotlib.pyplot as plt
OUT_DIR = Path(__file__).parent
rng = np.random.default_rng(20261101)
report = []
def check(name, condition):
report.append((name, bool(condition)))
print(f" [{'ok' if condition else 'FAIL'}] {name}")
Generate one nonlinear regression dataset and a dense test grid.
def truth(x):
return np.sin(1.7 * x) + 0.13 * x**2
n = 72
x_train = np.sort(rng.uniform(-3.0, 3.0, n))
y_train = truth(x_train) + rng.normal(0.0, 0.24, n)
x_grid = np.linspace(-3.0, 3.0, 500)
y_grid = truth(x_grid)
check("[B-data] the training set contains 72 feature-label pairs",
len(x_train) == len(y_train) == n)
[ok] [B-data] the training set contains 72 feature-label pairs
Define a regression-stump base learner: training data go in and a piecewise-constant learned hypothesis comes out.
def train_stump(x, y, min_leaf=5):
"""Return the squared-error-minimizing one-split regression stump."""
order = np.argsort(x)
xs, ys = x[order], y[order]
best = None
for split in range(min_leaf, len(xs) - min_leaf + 1):
if xs[split - 1] == xs[split]:
continue
left, right = ys[:split], ys[split:]
loss = ((left - left.mean()) ** 2).sum() \
+ ((right - right.mean()) ** 2).sum()
if best is None or loss < best[0]:
cut = (xs[split - 1] + xs[split]) / 2
best = (loss, cut, left.mean(), right.mean())
_, cut, left_value, right_value = best
def learned_hypothesis(z):
z = np.asarray(z)
return np.where(z <= cut, left_value, right_value)
return learned_hypothesis
one_stump = train_stump(x_train, y_train)
check("[B-base] the learned stump has two prediction values",
len(np.unique(one_stump(x_grid))) == 2)
[ok] [B-base] the learned stump has two prediction values
Train stumps independently on bootstrap samples and average them.
n_bag = 30
bag_hypotheses = []
for _ in range(n_bag):
sample = rng.integers(0, n, n)
bag_hypotheses.append(train_stump(x_train[sample], y_train[sample]))
bag_predictions = np.vstack([h(x_grid) for h in bag_hypotheses])
bagged = bag_predictions.mean(axis=0)
check("[B-bag] bagging averages 30 independently learned hypotheses",
bag_predictions.shape == (n_bag, len(x_grid)) and
np.allclose(bagged, bag_predictions.mean(axis=0)))
[ok] [B-bag] bagging averages 30 independently learned hypotheses
Train stumps sequentially to predict the current residual labels.
n_boost = 24
learn_rate = 0.28
initial = float(y_train.mean())
boost_train = np.full(n, initial)
boosted = np.full(len(x_grid), initial)
boost_hypotheses = []
boost_loss = [np.mean((y_train - boost_train) ** 2)]
for _ in range(n_boost):
residual_labels = y_train - boost_train
hypothesis = train_stump(x_train, residual_labels)
boost_hypotheses.append(hypothesis)
boost_train += learn_rate * hypothesis(x_train)
boosted += learn_rate * hypothesis(x_grid)
boost_loss.append(np.mean((y_train - boost_train) ** 2))
check("[B-boost] sequential residual updates lower training squared error",
boost_loss[-1] < 0.25 * boost_loss[0])
[ok] [B-boost] sequential residual updates lower training squared error
Build out-of-fold prediction features and learn their aggregation.
def train_polynomial(x, y, degree):
design = np.vander(x, degree + 1, increasing=True)
weights = np.linalg.lstsq(design, y, rcond=None)[0]
return lambda z: np.vander(np.asarray(z), degree + 1,
increasing=True) @ weights
base_algorithms = [lambda x, y: train_polynomial(x, y, 1),
lambda x, y: train_polynomial(x, y, 3),
train_stump]
n_folds = 6
folds = np.arange(n) % n_folds
oof = np.empty((n, len(base_algorithms)))
for fold in range(n_folds):
held_out = folds == fold
for j, algorithm in enumerate(base_algorithms):
hypothesis = algorithm(x_train[~held_out], y_train[~held_out])
oof[held_out, j] = hypothesis(x_train[held_out])
meta_design = np.column_stack((np.ones(n), oof))
meta_weights = np.linalg.lstsq(meta_design, y_train, rcond=None)[0]
full_hypotheses = [algorithm(x_train, y_train)
for algorithm in base_algorithms]
stack_features = np.column_stack([h(x_grid) for h in full_hypotheses])
stacked = np.column_stack((np.ones(len(x_grid)), stack_features)) @ meta_weights
check("[B-stack] each training row contains three out-of-fold predictions",
oof.shape == (n, 3) and np.all(np.isfinite(oof)))
check("[B-stack] the aggregation rule acts on base predictions as features",
meta_weights.shape == (4,) and np.all(np.isfinite(stacked)))
[ok] [B-stack] each training row contains three out-of-fold predictions [ok] [B-stack] the aggregation rule acts on base predictions as features wrote /Users/junga1/dictionaryappliedml/pythondemos/baselearner.png 6/6 checks pass.
Plot the three ensembles and write baselearner.png.
fig, axes = plt.subplots(1, 3, figsize=(10.5, 3.25), sharex=True, sharey=True)
ax = axes[0]
for prediction in bag_predictions[:8]:
ax.plot(x_grid, prediction, color="0.72", lw=0.7)
ax.plot(x_grid, bagged, color="black", lw=2.0)
ax.set_title("bagging\nparallel bootstrap samples")
ax = axes[1]
for k in (1, 4, 10):
partial = np.full(len(x_grid), initial)
for hypothesis in boost_hypotheses[:k]:
partial += learn_rate * hypothesis(x_grid)
ax.plot(x_grid, partial, color="0.68", lw=0.9)
ax.plot(x_grid, boosted, color="black", lw=2.0)
ax.set_title("boosting\nsequential residual labels")
ax = axes[2]
for hypothesis in full_hypotheses:
ax.plot(x_grid, hypothesis(x_grid), color="0.68", lw=0.9)
ax.plot(x_grid, stacked, color="black", lw=2.0)
ax.set_title("stacking\nbase predictions become features")
for ax in axes:
ax.scatter(x_train, y_train, s=11, facecolors="white",
edgecolors="#31688e", linewidths=0.7, zorder=4)
ax.plot(x_grid, y_grid, color="#b23a2b", ls="--", lw=1.2)
ax.set_xlabel(r"feature $x$")
axes[0].set_ylabel(r"label $y$")
fig.tight_layout()
fig.savefig(OUT_DIR / "baselearner.png", dpi=180)
print(f" wrote {OUT_DIR / 'baselearner.png'}")
n_fail = sum(not passed for _, passed in report)
print(f"\n{len(report) - n_fail}/{len(report)} checks pass.")
raise SystemExit(1 if n_fail else 0)
B-fig writes when the script runs