B-uniform writes when the script runsDictionary of Applied Machine Learning · training error
Updated on 2026-11-03
Numerical companion to the entry training error: it recomputes what the entry states and prints one line per check
The entry's weather narrative, carried out on the record itself: the daily minimum and maximum air temperature at Krems an der Donau for all 366 days of 2024 (GeoSphere Austria, station 3805, dataset klima-v2-1d). Each day is one data point: its feature is the morning minimum, its label the maximum of the day.
Run it without installing anything:uv run https://dictionaryofml.org/terms/trainerr.py
uv downloads this script and the pinned NumPy and Matplotlib it needs, then runs it; the script fetches any input file it uses. To keep the output files, download trainerr.py into a folder and run uv run trainerr.py there. With NumPy and Matplotlib already installed, python3 trainerr.py, from any directory — it writes its output files into the current directory. Fixed seeds, so the printed numbers reproduce exactly. Download trainerr.py · Notebook · Open in Colab
Before running a downloaded copy, scan it: uvx bandit -q -t B102,B307,B602,B605 trainerr.py (B102,B307,B602,B605 are the exec, eval and shell checks of bandit) reports nothing: the script runs no shell command and evaluates no generated code.
One cell per block of the script: the code, and what that code printed when it last ran here
"""
trainerr.py -- numerical companion to the entry 'training error'.
The entry's weather narrative, carried out on the record itself: the daily
minimum and maximum air temperature at Krems an der Donau for all 366 days
of 2024 (GeoSphere Austria, station 3805, dataset klima-v2-1d). Each day
is one data point: its feature is the morning minimum, its label the
maximum of the day.
Three examples. The first is the picture the entry opens with: the days
of five years, 2020 to 2024, as a scatterplot of morning minimum against
maximum, with three of them as the training set. Two hypotheses are
learned from those three days, the same pair as in the 'training set' and
'validation set' entries: the straight line h1 that ERM delivers over the
linear model, and the degree-two polynomial h2 through all three days.
The training error of h1 is the average of its three squared prediction
errors, 7.91; h2 interpolates, so its training error is zero, yet its
average loss over the five years is 42.90 against 24.22 for h1 -- the
smaller training error belongs to the worse hypothesis. Holding the
intercept of h1 fixed and sweeping its slope then shows the training error
and the risk as two functions of one parameter: the risk is the average
loss over all days of the five years, which stand in for the probability
distribution of a day, and its minimizer is a smaller slope than ERM's.
The second is larger: the first day of each month from January to
August, eight days, is the training set, and the remaining 358 days are
held back. A line learned from the eight days incurs an average squared
error of 20.36 on them and of 21.60 on the held-back days, a gap of six
percent: a line has two parameters, so there is little room to follow the
noise of eight days. Raising the degree of the polynomial widens the gap
without bound. The training error falls monotonically from 62.43 for the
constant to zero at degree seven, while the held-back error is smallest
at degree one, at 21.60, and reaches 23908 at degree seven, where the
polynomial swings wildly between the training days.
Deterministic: fixed seeds for the random draws of [B-def] and [B-uniform];
every fit is the closed-form least-squares solution, and the subsets of the
later examples are fixed strides through 2024.
Self-contained: numpy + matplotlib only (stdlib urllib for the download).
Blocks
------
[B-fetch] Download the daily minimum and maximum temperature at Krems
for 2020 to 2024; write the records to trainerr_weather.csv
and check them against the archive.
[B-opening] The entry's opening figure: the days of five years, three of
them the training set, and the two hypotheses learned from
them -- the ERM line h1 and the interpolating degree-two
polynomial h2. The training error of h1 is the average of
three squared prediction errors, that of h2 is zero, and h2
is the worse of the two over the five years; write the
points, both curves and the errors of h1.
[B-curves] The training error and the risk along the slice through h1 in
which the intercept is held fixed and the slope varies: two
parabolas with different minimizers. The ERM slope minimizes
the training error, and the training error of the ERM line
lies below its risk. Writes the CSV behind the entry's
second figure.
[B-def] The training error of the line learned from the eight
training days: the average of the eight squared errors, and
no line attains a smaller one.
[B-degree] Polynomials of degree 0 to 7 learned from the same eight days:
the training error never increases with the degree, while the
error on the 358 held-back days is smallest at a low degree
and then explodes.
[B-uniform] The training error as a noisy estimate of the risk, over 300
random draws of a training set of each size from the year.
For one fixed hypothesis, the least squares line of the
whole year, the deviation of its training error from its
risk averages to zero. For the hypothesis that ERM (least
squares over all affine maps) learns from the draw, the
deviation is negative on average. The largest deviation
over the hypothesis space, approximated by the maximum over
a grid of 2500 affine maps (50 slopes times 50 intercepts)
and the learned hypothesis, bounds the ERM gap in every draw
and shrinks as the training set grows. Writes the CSV behind
the entry's third figure.
Outputs
-------
trainerr_weather.csv : date, tmin, tmax of the 1827 downloaded days
trainerr_open_days.csv : tmin, tmax of the days of 2020 to 2024
trainerr_open_days_plot.csv : the 20 of them (every 92nd) the figure draws
trainerr_open_train.csv : tmin, tmax of the three training days
trainerr_open_line.csv : tmin, tmax along the line with the ERM slope
trainerr_open_err.csv : tmin, line, errplus, errminus -- the vertical
gap between each training day and the line
trainerr_curves.csv : slope, train, risk -- the two functions of the
slope, and trainerr_curves_min.csv marks their
minimizers
trainerr_degree.csv : degree, train, heldout -- the two average
losses of the polynomial of each degree
trainerr_uniform.csv : size, fixed_mean, fixed_spread, erm_mean,
erm_spread, sup_mean, sup_spread -- deviation of
the training error from the risk over 300 draws
trainerr.png : preview (checking only)
"""
# /// script
# requires-python = ">=3.10"
# dependencies = ["matplotlib==3.10.7", "numpy==2.3.4"]
# ///
import json
import urllib.request
from pathlib import Path
import numpy as np
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
OUT_DIR = Path(__file__).parent
report = []
def check(name, ok):
report.append((name, bool(ok)))
print(f" [{'ok' if ok else 'FAIL'}] {name}")
def avg_sqerr(x, y, coef):
return float(np.mean((y - np.polyval(coef, x)) ** 2))
Download the daily minimum and maximum temperature at Krems for 2020 to 2024; write the records to trainerr_weather.csv and check them against the archive.
print("[B-fetch] the record")
URL = ("https://dataset.api.hub.geosphere.at/v1/station/historical/"
"klima-v2-1d?parameters=tlmin,tlmax&station_ids=3805"
"&start=2020-01-01&end=2024-12-31")
with urllib.request.urlopen(URL, timeout=120) as resp:
payload = json.load(resp)
params = payload["features"][0]["properties"]["parameters"]
dates5 = [stamp[:10] for stamp in payload["timestamps"]]
tmin5 = np.array(params["tlmin"]["data"], dtype=float)
tmax5 = np.array(params["tlmax"]["data"], dtype=float)
with open(OUT_DIR / "trainerr_weather.csv", "w") as f:
f.write("date,tmin,tmax\n")
for day, lo, hi in zip(dates5, tmin5, tmax5):
f.write(f"{day},{lo},{hi}\n")
print(f" {len(dates5)} days, morning minimum from {tmin5.min():.1f} to "
f"{tmin5.max():.1f}, maximum from {tmax5.min():.1f} to {tmax5.max():.1f}")
check("[B-fetch] 1827 days downloaded for 2020 to 2024", len(dates5) == 1827)
check("[B-fetch] no day is missing a temperature",
not np.isnan(tmin5).any() and not np.isnan(tmax5).any())
in2024 = np.array([d.startswith("2024") for d in dates5])
dates = [d for d in dates5 if d.startswith("2024")] # the later blocks use 2024
tmin, tmax = tmin5[in2024], tmax5[in2024]
check("[B-fetch] the record matches the archive (2024-01-01: 2.6 to 9.6)",
len(dates) == 366 and (dates[0], tmin[0], tmax[0]) == ("2024-01-01", 2.6, 9.6))
[B-fetch] the record
1827 days, morning minimum from -13.5 to 22.0, maximum from -3.6 to 35.7
[ok] [B-fetch] 1827 days downloaded for 2020 to 2024
[ok] [B-fetch] no day is missing a temperature
[ok] [B-fetch] the record matches the archive (2024-01-01: 2.6 to 9.6)
The entry's opening figure: the days of five years, three of them the training set, and the two hypotheses learned from them -- the ERM line h1 and the interpolating degree-two polynomial h2. The training error of h1 is the average of three squared prediction errors, that of h2 is zero, and h2 is the worse of the two over the five years; write the points, both curves and the errors of h1.
print("\n[B-opening] the entry's opening figure")
# the two hypotheses of the 'training set' and 'validation set' entries,
# learned here from three days of the record: h1 over the linear model and
# h2 over the degree-two polynomials, which interpolates the three days
def pick_day(q, pct):
"""Among the days whose morning minimum lies within half a degree of the
q-quantile of all morning minima, the day at the pct-quantile of the
maxima: three days spread over the range, off the line by a few degrees."""
target = np.quantile(tmin5, q)
win = np.where(np.abs(tmin5 - target) <= 0.5)[0]
order = win[np.argsort(tmax5[win], kind="stable")]
return order[int(round(pct * (len(order) - 1)))]
three = np.array([pick_day(q, p) for q, p in ((0.12, 0.5), (0.55, 0.75), (0.90, 0.5))])
x3, y3 = tmin5[three], tmax5[three]
order3 = np.argsort(x3)
three, x3, y3 = three[order3], x3[order3], y3[order3]
coef1 = np.polyfit(x3, y3, 1) # h1, the line ERM delivers
coef2 = np.polyfit(x3, y3, 2) # h2, the interpolating parabola
open_trainerr = avg_sqerr(x3, y3, coef1)
open_trainerr2 = avg_sqerr(x3, y3, coef2)
open_risk = avg_sqerr(tmin5, tmax5, coef1)
open_risk2 = avg_sqerr(tmin5, tmax5, coef2)
for d, x, y in zip(three, x3, y3):
print(f" training day {dates5[d]}: morning minimum {x:5.1f}, "
f"maximum {y:5.1f}, h1 {np.polyval(coef1, x):5.1f}")
print(f" h1 = {coef1[1]:.2f} + {coef1[0]:.2f} x: training error "
f"{open_trainerr:.2f} on the three days, risk {open_risk:.2f} over the "
f"five years")
print(f" h2 = {coef2[2]:.2f} + {coef2[1]:.2f} x {coef2[0]:+.4f} x^2: "
f"training error {open_trainerr2:.1e}, risk {open_risk2:.2f}")
check("[B-opening] three training days spread over the range of morning minima",
x3.max() - x3.min() > 15.0)
check("[B-opening] the training error is the average of the three squared "
"prediction errors",
np.isclose(open_trainerr, np.mean((y3 - np.polyval(coef1, x3)) ** 2)))
gen0 = np.random.default_rng(3)
others0 = coef1 + gen0.normal(0.0, [0.08, 0.8], size=(4000, 2))
check("[B-opening] no other line has a smaller training error on the three "
"days: h1 is the ERM solution",
min(avg_sqerr(x3, y3, c) for c in others0) >= open_trainerr)
check("[B-opening] h2 passes through all three days, so its training error "
"is zero", open_trainerr2 < 1e-16)
check(f"[B-opening] yet h2 is the worse hypothesis over the five years: "
f"risk {open_risk2:.2f} against {open_risk:.2f} for h1",
open_risk2 > 1.5 * open_risk)
np.savetxt(OUT_DIR / "trainerr_open_days.csv", np.stack([tmin5, tmax5], 1),
delimiter=",", header="tmin,tmax", comments="", fmt="%.1f")
# the entry's figure draws every 92nd day, twenty of the 1827, so the dots
# do not crowd the three training days; every number stays on all days
PLOT_STRIDE = 92
plot_idx = np.arange(0, tmin5.size, PLOT_STRIDE)
np.savetxt(OUT_DIR / "trainerr_open_days_plot.csv",
np.stack([tmin5[plot_idx], tmax5[plot_idx]], 1),
delimiter=",", header="tmin,tmax", comments="", fmt="%.1f")
check("[B-opening] the figure draws twenty of the days, every 92nd",
plot_idx.size == 20)
np.savetxt(OUT_DIR / "trainerr_open_train.csv", np.stack([x3, y3], 1),
delimiter=",", header="tmin,tmax", comments="", fmt="%.1f")
xs_line = np.array([tmin5.min() - 1.0, tmin5.max() + 1.0])
np.savetxt(OUT_DIR / "trainerr_open_line.csv",
np.stack([xs_line, np.polyval(coef1, xs_line)], 1), delimiter=",",
header="tmin,tmax", comments="", fmt="%.3f")
xs_poly = np.linspace(xs_line[0], xs_line[1], 200)
np.savetxt(OUT_DIR / "trainerr_open_poly.csv",
np.stack([xs_poly, np.polyval(coef2, xs_poly)], 1), delimiter=",",
header="tmin,tmax", comments="", fmt="%.3f")
with open(OUT_DIR / "trainerr_open_err.csv", "w") as f:
f.write("tmin,line,errplus,errminus\n")
for x, y in zip(x3, y3):
on_line = float(np.polyval(coef1, x))
gap = y - on_line
f.write(f"{x:.2f},{on_line:.3f},{max(gap, 0.0):.3f},{max(-gap, 0.0):.3f}\n")
[B-opening] the entry's opening figure
training day 2023-11-19: morning minimum -1.6, maximum 6.3, h1 8.3
training day 2020-05-07: morning minimum 7.4, maximum 23.2, h1 19.2
training day 2024-08-27: morning minimum 16.1, maximum 27.8, h1 29.8
h1 = 10.20 + 1.22 x: training error 7.91 on the three days, risk 24.22 over the five years
h2 = 10.21 + 2.32 x -0.0762 x^2: training error 1.3e-28, risk 42.90
[ok] [B-opening] three training days spread over the range of morning minima
[ok] [B-opening] the training error is the average of the three squared prediction errors
[ok] [B-opening] no other line has a smaller training error on the three days: h1 is the ERM solution
[ok] [B-opening] h2 passes through all three days, so its training error is zero
[ok] [B-opening] yet h2 is the worse hypothesis over the five years: risk 42.90 against 24.22 for h1
[ok] [B-opening] the figure draws twenty of the days, every 92nd
The training error and the risk along the slice through h1 in which the intercept is held fixed and the slope varies: two parabolas with different minimizers. The ERM slope minimizes the training error, and the training error of the ERM line lies below its risk. Writes the CSV behind the entry's second figure.
print("\n[B-curves] the two functions of the slope")
# one-dimensional slice through h1: the intercept is held at its ERM value
# and only the slope varies, so both errors are functions of one number
W0 = float(coef1[1])
def avg_sqerr_slope(x, y, w):
return float(np.mean((y - (W0 + w * x)) ** 2))
def slope_erm(x, y):
"""The slope minimizing the average squared error of W0 + w x on the
given days, in closed form: the ERM solution along the slice."""
return float(np.sum(x * (y - W0)) / np.sum(x ** 2))
w_erm = float(coef1[0])
slopes = np.linspace(0.6, 1.5, 181)
train_of_w = np.array([avg_sqerr_slope(x3, y3, w) for w in slopes])
risk_of_w = np.array([avg_sqerr_slope(tmin5, tmax5, w) for w in slopes])
w_risk = slope_erm(tmin5, tmax5) # minimizer of the risk
print(f" the training error is smallest at slope {w_erm:.3f} "
f"({open_trainerr:.2f}), the risk at slope {w_risk:.3f} "
f"({avg_sqerr_slope(tmin5, tmax5, w_risk):.2f}); at the ERM slope the "
f"risk is {open_risk:.2f}")
check("[B-curves] on the grid the training error is smallest at the ERM slope",
abs(slopes[np.argmin(train_of_w)] - w_erm) <= 0.005)
check(f"[B-curves] the risk is smallest at a smaller slope "
f"({w_risk:.3f} against {w_erm:.3f})",
w_risk < w_erm - 0.01)
check("[B-curves] the training error of the ERM line is below its risk",
open_trainerr < open_risk)
check("[B-curves] the risk is above the training error on the ERM side of "
"the risk minimizer and the two curves cross",
(train_of_w < risk_of_w).any() and (train_of_w > risk_of_w).any())
with open(OUT_DIR / "trainerr_curves.csv", "w") as f:
f.write("slope,train,risk\n")
for w, t, r in zip(slopes, train_of_w, risk_of_w):
f.write(f"{w:.3f},{t:.4f},{r:.4f}\n")
with open(OUT_DIR / "trainerr_curves_min.csv", "w") as f:
f.write("slope,train,risk\n")
f.write(f"{w_erm:.4f},{open_trainerr:.4f},{open_risk:.4f}\n")
f.write(f"{w_risk:.4f},{avg_sqerr_slope(x3, y3, w_risk):.4f},"
f"{avg_sqerr_slope(tmin5, tmax5, w_risk):.4f}\n")
[B-curves] the two functions of the slope
the training error is smallest at slope 1.218 (7.91), the risk at slope 1.086 (22.63); at the ERM slope the risk is 24.22
[ok] [B-curves] on the grid the training error is smallest at the ERM slope
[ok] [B-curves] the risk is smallest at a smaller slope (1.086 against 1.218)
[ok] [B-curves] the training error of the ERM line is below its risk
[ok] [B-curves] the risk is above the training error on the ERM side of the risk minimizer and the two curves cross
The training error of the line learned from the eight training days: the average of the eight squared errors, and no line attains a smaller one.
print("\n[B-def] the training error of a line learned by ERM")
train = np.array([0, 31, 60, 91, 121, 152, 182, 213]) # the first day of January to August 2024 (leap year)
held = np.setdiff1d(np.arange(len(tmin)), train)
line = np.polyfit(tmin[train], tmax[train], 1)
losses = (tmax[train] - np.polyval(line, tmin[train])) ** 2
trainerr = avg_sqerr(tmin[train], tmax[train], line)
print(f" line: maximum = {line[0]:.3f} * minimum + {line[1]:.3f}")
print(f" training error {trainerr:.2f} = average of {len(train)} squared "
f"errors (smallest {losses.min():.2f}, largest {losses.max():.2f}); "
f"on the {len(held)} held-back days {avg_sqerr(tmin[held], tmax[held], line):.2f}")
check("[B-def] the training error is the average of the per-day losses",
np.isclose(trainerr, float(np.mean(losses)), atol=1e-12))
gen = np.random.default_rng(1)
others = line + gen.normal(0.0, [0.05, 0.5], size=(2000, 2))
worse = min(avg_sqerr(tmin[train], tmax[train], c) for c in others)
print(f" 2000 other lines (slope and offset perturbed): smallest training "
f"error {worse:.2f}")
check("[B-def] no other line has a smaller training error: ERM delivers the "
"minimum", worse >= trainerr)
[B-def] the training error of a line learned by ERM
line: maximum = 0.901 * minimum + 11.975
training error 20.36 = average of 8 squared errors (smallest 0.09, largest 76.74); on the 358 held-back days 21.60
[ok] [B-def] the training error is the average of the per-day losses
2000 other lines (slope and offset perturbed): smallest training error 20.36
[ok] [B-def] no other line has a smaller training error: ERM delivers the minimum
Polynomials of degree 0 to 7 learned from the same eight days: the training error never increases with the degree, while the error on the 358 held-back days is smallest at a low degree and then explodes.
print("\n[B-degree] the two errors against the degree of the polynomial")
DEGREES = list(range(0, 8))
rows = []
for d in DEGREES:
coef = np.polyfit(tmin[train], tmax[train], d)
rows.append((d, avg_sqerr(tmin[train], tmax[train], coef),
avg_sqerr(tmin[held], tmax[held], coef)))
print(f" degree {d:2d}: training error {rows[-1][1]:8.2f}, "
f"held-back error {rows[-1][2]:12.2f}")
train_curve = [r[1] for r in rows]
held_curve = [r[2] for r in rows]
check("[B-degree] the training error never increases with the degree",
all(a >= b - 1e-9 for a, b in zip(train_curve, train_curve[1:])))
check("[B-degree] the training error is below the held-back error at every "
"degree", all(r[1] < r[2] for r in rows))
best = int(np.argmin(held_curve))
print(f" smallest held-back error at degree {best}; at degree {DEGREES[-1]} the "
f"training error is {train_curve[-1]:.2f} and the held-back error "
f"{held_curve[-1]:.0f}")
check("[B-degree] the held-back error is smallest at a low degree and grows "
"for larger degrees",
best <= 3 and held_curve[-1] > 10.0 * min(held_curve))
with open(OUT_DIR / "trainerr_degree.csv", "w") as f:
f.write("degree,train,heldout\n")
for d, t, h in rows:
f.write(f"{d},{t:.4f},{h:.4f}\n")
[B-degree] the two errors against the degree of the polynomial
degree 0: training error 62.43, held-back error 88.80
degree 1: training error 20.36, held-back error 21.60
degree 2: training error 20.30, held-back error 21.85
degree 3: training error 19.70, held-back error 24.08
degree 4: training error 15.12, held-back error 353.29
degree 5: training error 6.34, held-back error 7534.06
degree 6: training error 0.09, held-back error 130636.71
degree 7: training error 0.00, held-back error 23908.03
[ok] [B-degree] the training error never increases with the degree
[ok] [B-degree] the training error is below the held-back error at every degree
smallest held-back error at degree 1; at degree 7 the training error is 0.00 and the held-back error 23908
[ok] [B-degree] the held-back error is smallest at a low degree and grows for larger degrees
The training error as a noisy estimate of the risk, over 300 random draws of a training set of each size from the year. For one fixed hypothesis, the least squares line of the whole year, the deviation of its training error from its risk averages to zero. For the hypothesis that ERM (least squares over all affine maps) learns from the draw, the deviation is negative on average. The largest deviation over the hypothesis space, approximated by the maximum over a grid of 2500 affine maps (50 slopes times 50 intercepts) and the learned hypothesis, bounds the ERM gap in every draw and shrinks as the training set grows. Writes the CSV behind the entry's third figure.
print("\n[B-uniform] one fixed hypothesis, the ERM hypothesis, and the largest deviation")
# the 366 days stand in for the probability distribution of a day: the
# "risk" of a line is its average loss over all of them, and a training set
# is a random subset of the year, drawn without replacement
pop_line = np.polyfit(tmin, tmax, 1) # the fixed hypothesis: least squares on the whole year, the smallest risk
SLOPES = np.linspace(0.8, 1.3, 50)
OFFSETS = np.linspace(6.0, 15.0, 50)
grid_w, grid_c = np.meshgrid(SLOPES, OFFSETS)
grid_w, grid_c = grid_w.ravel(), grid_c.ravel() # 2500 affine maps h(x) = w x + c, a grid over the hypothesis space
loss_grid = (tmax[None, :] - (grid_w[:, None] * tmin[None, :] + grid_c[:, None])) ** 2
loss_fixed = (tmax - np.polyval(pop_line, tmin)) ** 2
risk_grid = loss_grid.mean(axis=1) # risk of every line of the grid
risk_fixed = float(loss_fixed.mean())
SIZES_U = [10, 20, 40, 80]
DRAWS_U = 300
gen_u = np.random.default_rng(2)
urows = []
for size in SIZES_U:
dev_fixed, dev_erm, dev_sup = [], [], []
for _ in range(DRAWS_U):
sub = gen_u.choice(len(tmin), size, replace=False)
emp_grid = loss_grid[:, sub].mean(axis=1) # training error of every grid hypothesis
erm = np.polyfit(tmin[sub], tmax[sub], 1) # ERM over all affine maps: least squares
loss_erm = (tmax - np.polyval(erm, tmin)) ** 2
gap_erm = float(loss_erm[sub].mean() - loss_erm.mean())
dev_fixed.append(float(loss_fixed[sub].mean()) - risk_fixed)
dev_erm.append(gap_erm)
# the largest deviation over the hypothesis space, approximated by the maximum
# over the grid and the learned hypothesis
dev_sup.append(max(float(np.max(np.abs(emp_grid - risk_grid))), abs(gap_erm)))
urows.append((size, np.mean(dev_fixed), np.std(dev_fixed),
np.mean(dev_erm), np.std(dev_erm),
np.mean(dev_sup), np.std(dev_sup),
np.mean(np.array(dev_erm) < 0.0)))
print(f" {size:>3} training days: fixed hypothesis {urows[-1][1]:+6.2f} "
f"(spread {urows[-1][2]:5.2f}), ERM hypothesis {urows[-1][3]:+6.2f} "
f"(spread {urows[-1][4]:5.2f}, below the risk in "
f"{100 * urows[-1][7]:.0f}% of the draws), uniform deviation "
f"{urows[-1][5]:6.2f} (spread {urows[-1][6]:5.2f})")
check("[B-uniform] for the fixed hypothesis the deviation averages to zero: each "
"mean is within three standard errors of zero",
all(abs(r[1]) < 3.0 * r[2] / np.sqrt(DRAWS_U) for r in urows))
check("[B-uniform] for the ERM hypothesis the deviation is negative on average: "
"each mean is below zero by more than three standard errors",
all(r[3] < -3.0 * r[4] / np.sqrt(DRAWS_U) for r in urows))
check("[B-uniform] the ERM gap is bounded by the uniform deviation in every "
"draw, and the uniform deviation exceeds the spread of the fixed hypothesis "
"at every size", all(r[5] > r[2] and -r[3] < r[5] for r in urows))
check("[B-uniform] the uniform deviation and the ERM gap both shrink as the "
"training set grows",
all(a[5] > b[5] and a[3] < b[3] for a, b in zip(urows, urows[1:])))
with open(OUT_DIR / "trainerr_uniform.csv", "w") as f:
f.write("size,fixed_mean,fixed_spread,erm_mean,erm_spread,sup_mean,sup_spread\n")
for r in urows:
f.write(f"{r[0]}," + ",".join(f"{v:.4f}" for v in r[1:7]) + "\n")
# ---- preview (checking only)
fig, axes = plt.subplots(2, 2, figsize=(12, 8.0))
axes = axes.ravel()
ax = axes[0]
ax.plot(tmin5, tmax5, ".", color="0.7", ms=3, label="days of 2020 to 2024")
ax.plot(xs_line, np.polyval(coef1, xs_line), "k-", lw=1.4,
label=f"h1, the ERM line (training error {open_trainerr:.2f})")
ax.plot(xs_poly, np.polyval(coef2, xs_poly), "k--", lw=1.4,
label="h2, the interpolating parabola (training error 0)")
for x, y in zip(x3, y3):
ax.plot([x, x], [float(np.polyval(coef1, x)), y], "k:", lw=1.0)
ax.plot(x3, y3, "o", color="black", ms=8, label="training set (3 days)")
ax.set_ylim(tmax5.min() - 4.0, tmax5.max() + 4.0)
ax.set_xlabel("morning minimum temperature (feature, °C)")
ax.set_ylabel("maximum temperature (label, °C)")
ax.set_title(f"Five years at Krems: training error {open_trainerr:.2f} for "
f"h1 and 0 for h2, risk {open_risk:.2f} against "
f"{open_risk2:.2f}", fontsize=9)
ax.legend(frameon=False, fontsize=8, loc="upper left")
ax = axes[1]
ax.plot(slopes, train_of_w, "k-", lw=1.4, label="training error (3 days)")
ax.plot(slopes, risk_of_w, "--", color="0.4", lw=1.4, label="risk (all days)")
ax.plot([w_erm], [open_trainerr], "o", color="black", ms=6,
label="ERM slope")
ax.plot([w_risk], [avg_sqerr_slope(tmin5, tmax5, w_risk)], "s", mfc="none",
mec="black", ms=7, label="risk minimizer")
ax.set_xlabel("slope w of the line")
ax.set_ylabel("average squared error")
ax.set_title("Training error and risk as functions of the hypothesis", fontsize=9)
ax.legend(frameon=False, fontsize=8)
ax = axes[2]
ax.semilogy(DEGREES, train_curve, "o-", color="black", ms=4,
label="training error, 40 days")
ax.semilogy(DEGREES, held_curve, "s--", color="0.4", ms=4,
label="error on the 358 held-back days")
ax.set_xlabel("degree of the polynomial")
ax.set_ylabel("average squared error")
ax.set_title("Learned by ERM from the same eight days", fontsize=9)
ax.legend(frameon=False, fontsize=8)
ax = axes[3]
xs = np.array(SIZES_U, dtype=float)
ax.errorbar(xs * 0.92, [r[1] for r in urows], yerr=[r[2] for r in urows],
fmt="o", ms=5, color="black", capsize=3, label="fixed line")
ax.errorbar(xs, [r[3] for r in urows], yerr=[r[4] for r in urows],
fmt="s", ms=5, color="0.4", capsize=3, label="line picked by ERM")
ax.errorbar(xs * 1.08, [r[5] for r in urows], yerr=[r[6] for r in urows],
fmt="^", ms=6, mfc="none", color="black", capsize=3,
label="largest deviation over 2500 lines")
ax.axhline(0.0, ls=":", color="0.6")
ax.set_xscale("log")
ax.set_xticks(SIZES_U)
ax.set_xticklabels([str(n) for n in SIZES_U])
ax.set_xlabel("number of training days")
ax.set_ylabel("training error minus risk")
ax.set_title("ERM's training error sits below its risk; the worst line "
"bounds the gap", fontsize=9)
ax.legend(frameon=False, fontsize=8)
fig.tight_layout()
fig.savefig(OUT_DIR / "trainerr.png", dpi=150)
plt.close(fig)
passed = sum(1 for _, ok in report if ok)
print(f"\n{passed}/{len(report)} checks pass")
[B-uniform] one fixed hypothesis, the ERM hypothesis, and the largest deviation
10 training days: fixed hypothesis +0.75 (spread 8.07), ERM hypothesis -7.51 (spread 9.66, below the risk in 81% of the draws), uniform deviation 25.07 (spread 12.06)
20 training days: fixed hypothesis -0.41 (spread 5.20), ERM hypothesis -3.76 (spread 5.23, below the risk in 74% of the draws), uniform deviation 16.83 (spread 7.61)
40 training days: fixed hypothesis +0.09 (spread 3.50), ERM hypothesis -1.40 (spread 3.54, below the risk in 67% of the draws), uniform deviation 11.61 (spread 5.99)
80 training days: fixed hypothesis -0.01 (spread 2.33), ERM hypothesis -0.74 (spread 2.35, below the risk in 66% of the draws), uniform deviation 7.87 (spread 3.55)
[ok] [B-uniform] for the fixed hypothesis the deviation averages to zero: each mean is within three standard errors of zero
[ok] [B-uniform] for the ERM hypothesis the deviation is negative on average: each mean is below zero by more than three standard errors
[ok] [B-uniform] the ERM gap is bounded by the uniform deviation in every draw, and the uniform deviation exceeds the spread of the fixed hypothesis at every size
[ok] [B-uniform] the uniform deviation and the ERM gap both shrink as the training set grows
22/22 checks pass
B-uniform writes when the script runs