B-cond writes when the script runsDictionary of Applied Machine Learning · validation error
Updated on 2026-11-03
Numerical companion to the entry validation error: it recomputes what the entry states and prints one line per check
The entry's weather narrative, carried out on the record itself: the daily minimum and maximum air temperature at Krems an der Donau for all 366 days of 2024 (GeoSphere Austria, station 3805, dataset klima-v2-1d). Each day is one data point: its feature is the morning minimum, its label the maximum of the day.
Run it without installing anything:uv run https://dictionaryofml.org/terms/valerr.py
uv downloads this script and the pinned NumPy and Matplotlib it needs, then runs it; the script fetches any input file it uses. To keep the output files, download valerr.py into a folder and run uv run valerr.py there. With NumPy and Matplotlib already installed, python3 valerr.py, from any directory — it writes its output files into the current directory. Fixed seeds, so the printed numbers reproduce exactly. Download valerr.py · Notebook · Open in Colab
Before running a downloaded copy, scan it: uvx bandit -q -t B102,B307,B602,B605 valerr.py (B102,B307,B602,B605 are the exec, eval and shell checks of bandit) reports nothing: the script runs no shell command and evaluates no generated code.
One cell per block of the script: the code, and what that code printed when it last ran here
"""
valerr.py -- numerical companion to the entry 'validation error'.
The entry's weather narrative, carried out on the record itself: the daily
minimum and maximum air temperature at Krems an der Donau for all 366 days
of 2024 (GeoSphere Austria, station 3805, dataset klima-v2-1d). Each day
is one data point: its feature is the morning minimum, its label the
maximum of the day.
The picture the entry opens with shows the days of five years, 2020 to
2024, as a scatterplot of morning minimum against maximum. Three of them
are the training set -- the same three as in the 'training error' entry --
and the hypothesis is h2, the degree-two polynomial through all three,
the second of the two hypotheses of the 'training set' and 'validation
set' entries. It interpolates, so its training error is zero. Three
further days are the validation set, and the average of the polynomial's
squared prediction errors on them is the validation error, 42.58, which
lands on its average loss over all 1827 days, 42.90: the number the
training error could not supply.
The larger example splits the year: every ninth day, forty days, is the
training set; twenty of the remaining days are the validation set; the
other 306 stay held back and stand in for the risk. The straight line is
h(x) = 10.47 + 1.05 x, and the three numbers are 18.96, 18.59 and 19.41. The validation error sits
near the risk while the training error is below it, but for a line the
gap is 0.45 and a twenty-day average fluctuates by about 5.4, so one
validation set cannot resolve it. Raising the degree makes the gap plain:
at degree five the three numbers are 17.93, 20.84 and 21.45, and at degree
nine they are 13.80, 35.12 and 228.96.
Finally the fluctuation itself: the validation error is an average of
per-day losses, so it is a random quantity. Recomputed over 300 draws of
a validation set of each size from the held-back days, its mean stays near
the risk while its spread falls from 11.36 at five days to 2.26 at eighty,
close to the standard deviation of a single day's loss divided by the
square root of the size, corrected for drawing without replacement.
Deterministic: fixed seed for the draws; every fit is the closed-form
least-squares solution and the subsets of the later examples are fixed
strides through 2024.
Self-contained: numpy + matplotlib only (stdlib urllib for the download).
Blocks
------
[B-fetch] Download the daily minimum and maximum temperature at Krems
for 2020 to 2024; write the records to valerr_weather.csv and
check them against the archive.
[B-opening] The entry's opening figure: the days of five years, three of
them the training set, the interpolating degree-two
polynomial h2 learned from them (training error zero), and
three validation days. The validation error is the average
of the polynomial's three squared prediction errors and
lands on its average loss over the five years; write the
points, the curve and the errors for the figure.
[B-def] The validation error of the line fitted to the forty training
days: the average of twenty squared errors, with the line
fixed. Prints the line itself and checks it against the
closed-form least-squares solution, so it is visibly the ERM
solution.
[B-risk] The validation error against the risk, for degrees 1, 5 and 9:
the training error is below the risk at every degree, and the
gap grows with the degree, while for the line it is smaller
than the fluctuation of a twenty-day average. The risk here
is the average loss over the 306 held-back days, and the
block prints how tightly those 306 days pin it.
[B-spread] The validation error recomputed over 300 draws of a validation
set of each size: the spread shrinks as the set grows. Writes
the CSV behind the entry's second figure.
[B-cond] The learned hypothesis is itself random: over 300 draws of a
training set of forty days the ERM quadratic changes every
time. The validation error on twenty days
drawn independently of the training set still deviates from
the risk of that draw's hypothesis by zero on average, while
a validation set that shares ten days with the training set,
and the training error itself, sit below it.
Outputs
-------
valerr_weather.csv : date, tmin, tmax of the 1827 downloaded days
valerr_open_days.csv : tmin, tmax of the days of 2020 to 2024
valerr_open_days_plot.csv : the 20 of them (every 92nd) the figure draws
valerr_open_train.csv : tmin, tmax of the three training days
valerr_open_val.csv : tmin, tmax of the five validation days
valerr_open_tree.csv : tmin, tmax along the step function of the tree
valerr_open_err.csv : tmin, tree, errplus, errminus -- the vertical
gap between each validation day and the tree
valerr_spread.csv : size, mean, spread of the validation error over
300 draws of a validation set of that size
valerr_risk.csv : size, risk -- the reference level for the figure
valerr_cond.csv : estimate, mean, spread -- deviation from the risk
of the learned hypothesis over 300 draws
valerr.png : preview (checking only; also shown on screen
when the demo is run from a terminal or a
notebook)
"""
# /// script
# requires-python = ">=3.10"
# dependencies = ["matplotlib==3.10.7", "numpy==2.3.4"]
# ///
import json
import sys
import urllib.request
from pathlib import Path
import numpy as np
import matplotlib
# Run headless by default: the Agg backend needs no display and keeps the
# saved preview byte-identical across runs, which the committed PNG relies
# on. When the demo is run from a terminal or inside a notebook, the
# default backend is kept instead, so the figures are shown as well as
# written.
INTERACTIVE = sys.stdout.isatty() or "ipykernel" in sys.modules
if not INTERACTIVE:
matplotlib.use("Agg")
import matplotlib.pyplot as plt
OUT_DIR = Path(__file__).parent
report = []
def check(name, ok):
report.append((name, bool(ok)))
print(f" [{'ok' if ok else 'FAIL'}] {name}")
def avg_sqerr(x, y, coef):
return float(np.mean((y - np.polyval(coef, x)) ** 2))
Download the daily minimum and maximum temperature at Krems for 2020 to 2024; write the records to valerr_weather.csv and check them against the archive.
print("[B-fetch] the record")
URL = ("https://dataset.api.hub.geosphere.at/v1/station/historical/"
"klima-v2-1d?parameters=tlmin,tlmax&station_ids=3805"
"&start=2020-01-01&end=2024-12-31")
with urllib.request.urlopen(URL, timeout=120) as resp:
payload = json.load(resp)
params = payload["features"][0]["properties"]["parameters"]
dates5 = [stamp[:10] for stamp in payload["timestamps"]]
tmin5 = np.array(params["tlmin"]["data"], dtype=float)
tmax5 = np.array(params["tlmax"]["data"], dtype=float)
with open(OUT_DIR / "valerr_weather.csv", "w") as f:
f.write("date,tmin,tmax\n")
for day, lo, hi in zip(dates5, tmin5, tmax5):
f.write(f"{day},{lo},{hi}\n")
print(f" {len(dates5)} days, morning minimum from {tmin5.min():.1f} to "
f"{tmin5.max():.1f}, maximum from {tmax5.min():.1f} to {tmax5.max():.1f}")
check("[B-fetch] 1827 days downloaded for 2020 to 2024", len(dates5) == 1827)
check("[B-fetch] no day is missing a temperature",
not np.isnan(tmin5).any() and not np.isnan(tmax5).any())
in2024 = np.array([d.startswith("2024") for d in dates5])
dates = [d for d in dates5 if d.startswith("2024")] # the later blocks use 2024
tmin, tmax = tmin5[in2024], tmax5[in2024]
check("[B-fetch] the record matches the archive (2024-01-01: 2.6 to 9.6)",
len(dates) == 366 and (dates[0], tmin[0], tmax[0]) == ("2024-01-01", 2.6, 9.6))
[B-fetch] the record
1827 days, morning minimum from -13.5 to 22.0, maximum from -3.6 to 35.7
[ok] [B-fetch] 1827 days downloaded for 2020 to 2024
[ok] [B-fetch] no day is missing a temperature
[ok] [B-fetch] the record matches the archive (2024-01-01: 2.6 to 9.6)
The entry's opening figure: the days of five years, three of them the training set, the interpolating degree-two polynomial h2 learned from them (training error zero), and three validation days. The validation error is the average of the polynomial's three squared prediction errors and lands on its average loss over the five years; write the points, the curve and the errors for the figure.
print("\n[B-opening] the entry's opening figure")
def pick_day(q, pct):
"""Among the days whose morning minimum lies within half a degree of the
q-quantile of all morning minima, the day at the pct-quantile of the
maxima (the same three training days as in trainerr.py)."""
target = np.quantile(tmin5, q)
win = np.where(np.abs(tmin5 - target) <= 0.5)[0]
order = win[np.argsort(tmax5[win], kind="stable")]
return order[int(round(pct * (len(order) - 1)))]
three = np.array([pick_day(q, p) for q, p in ((0.12, 0.5), (0.55, 0.75), (0.90, 0.5))])
threeval = np.array([pick_day(q, 0.5) for q in (0.04, 0.48, 0.96)])
x3, y3 = tmin5[three], tmax5[three]
xv, yv = tmin5[threeval], tmax5[threeval]
order3 = np.argsort(x3)
x3, y3 = x3[order3], y3[order3]
orderv = np.argsort(xv)
xv, yv = xv[orderv], yv[orderv]
coef2 = np.polyfit(x3, y3, 2) # h2, the interpolating parabola
coef1 = np.polyfit(x3, y3, 1) # h1, kept for the contrast only
open_trainerr = avg_sqerr(x3, y3, coef2)
open_valerr = avg_sqerr(xv, yv, coef2)
open_risk = avg_sqerr(tmin5, tmax5, coef2)
for label, xs, ys in (("training", x3, y3), ("validation", xv, yv)):
for x, y in zip(xs, ys):
print(f" {label:>10} day: morning minimum {x:5.1f}, maximum {y:5.1f}, "
f"h2 {float(np.polyval(coef2, x)):6.1f}")
print(f" h2 = {coef2[2]:.2f} + {coef2[1]:.2f} x {coef2[0]:+.4f} x^2; "
f"training error {open_trainerr:.1e}, validation error "
f"{open_valerr:.2f}, risk {open_risk:.2f} over the five years")
print(f" for the contrast, h1 = {coef1[1]:.2f} + {coef1[0]:.2f} x has "
f"training error {avg_sqerr(x3, y3, coef1):.2f} and validation error "
f"{avg_sqerr(xv, yv, coef1):.2f}")
check("[B-opening] h2 passes through the three training days: training "
"error zero", open_trainerr < 1e-16)
check("[B-opening] no validation day is a training day",
not set(threeval) & set(three))
check("[B-opening] the validation error is the average of the three squared "
"prediction errors",
np.isclose(open_valerr, np.mean((yv - np.polyval(coef2, xv)) ** 2)))
check(f"[B-opening] the validation error {open_valerr:.2f} is large although "
f"the training error is zero, and it lands within a tenth of the risk "
f"{open_risk:.2f}",
open_valerr > 10.0 and abs(open_valerr - open_risk) < 0.1 * open_risk)
check("[B-opening] the zero training error hides that h2 is the worse of "
"the two hypotheses on the validation days",
avg_sqerr(xv, yv, coef2) > 3.0 * avg_sqerr(xv, yv, coef1))
np.savetxt(OUT_DIR / "valerr_open_days.csv", np.stack([tmin5, tmax5], 1),
delimiter=",", header="tmin,tmax", comments="", fmt="%.1f")
# the entry's figure draws every 92nd day, twenty of the 1827; the numbers
# of the entry stay on all days
PLOT_STRIDE = 92
plot_idx = np.arange(0, tmin5.size, PLOT_STRIDE)
np.savetxt(OUT_DIR / "valerr_open_days_plot.csv",
np.stack([tmin5[plot_idx], tmax5[plot_idx]], 1),
delimiter=",", header="tmin,tmax", comments="", fmt="%.1f")
check("[B-opening] the figure draws twenty of the days, every 92nd",
plot_idx.size == 20)
np.savetxt(OUT_DIR / "valerr_open_train.csv", np.stack([x3, y3], 1),
delimiter=",", header="tmin,tmax", comments="", fmt="%.1f")
np.savetxt(OUT_DIR / "valerr_open_val.csv", np.stack([xv, yv], 1),
delimiter=",", header="tmin,tmax", comments="", fmt="%.1f")
lo, hi = tmin5.min() - 1.0, tmin5.max() + 1.0
xs_poly = np.linspace(lo, hi, 200)
np.savetxt(OUT_DIR / "valerr_open_poly.csv",
np.stack([xs_poly, np.polyval(coef2, xs_poly)], 1), delimiter=",",
header="tmin,tmax", comments="", fmt="%.3f")
with open(OUT_DIR / "valerr_open_err.csv", "w") as f:
f.write("tmin,poly,errplus,errminus\n")
for x, y in zip(xv, yv):
on_poly = float(np.polyval(coef2, x))
gap = y - on_poly
f.write(f"{x:.2f},{on_poly:.3f},{max(gap, 0.0):.3f},{max(-gap, 0.0):.3f}\n")
[B-opening] the entry's opening figure
training day: morning minimum -1.6, maximum 6.3, h2 6.3
training day: morning minimum 7.4, maximum 23.2, h2 23.2
training day: morning minimum 16.1, maximum 27.8, h2 27.8
validation day: morning minimum -5.1, maximum 6.7, h2 -3.6
validation day: morning minimum 5.6, maximum 16.3, h2 20.8
validation day: morning minimum 17.9, maximum 28.4, h2 27.3
h2 = 10.21 + 2.32 x -0.0762 x^2; training error 1.3e-28, validation error 42.58, risk 42.90 over the five years
for the contrast, h1 = 10.20 + 1.22 x has training error 7.91 and validation error 6.98
[ok] [B-opening] h2 passes through the three training days: training error zero
[ok] [B-opening] no validation day is a training day
[ok] [B-opening] the validation error is the average of the three squared prediction errors
[ok] [B-opening] the validation error 42.58 is large although the training error is zero, and it lands within a tenth of the risk 42.90
[ok] [B-opening] the zero training error hides that h2 is the worse of the two hypotheses on the validation days
[ok] [B-opening] the figure draws twenty of the days, every 92nd
The validation error of the line fitted to the forty training days: the average of twenty squared errors, with the line fixed. Prints the line itself and checks it against the closed-form least-squares solution, so it is visibly the ERM solution.
print("\n[B-def] the validation error of a fitted line")
idx = np.arange(len(tmin))
train = idx[::9][:40] # every ninth day, forty of them
rest = np.setdiff1d(idx, train)
val = rest[::16][:20] # twenty of the remaining days
held = np.setdiff1d(rest, val) # the other 306, standing in for the risk
line = np.polyfit(tmin[train], tmax[train], 1)
losses = (tmax[val] - np.polyval(line, tmin[val])) ** 2
valerr = avg_sqerr(tmin[val], tmax[val], line)
print(f" {len(train)} training days, {len(val)} validation days, "
f"{len(held)} held back")
print(f" the line is h(x) = {line[1]:.2f} + {line[0]:.2f} x, the "
f"least-squares fit to the {len(train)} training days")
# the same line computed in closed form, to show that polyfit is ERM with
# the squared error loss over the linear model and nothing else
design = np.column_stack([tmin[train], np.ones(len(train))])
closed = np.linalg.solve(design.T @ design, design.T @ tmax[train])
check("[B-def] the line is the ERM solution: polyfit matches the closed-form "
"least-squares solution", np.allclose(line, closed))
print(f" validation error {valerr:.2f} = average of {len(val)} squared "
f"errors (smallest {losses.min():.2f}, largest {losses.max():.2f})")
check("[B-def] the validation error is the average of the per-day losses",
np.isclose(valerr, float(np.mean(losses)), atol=1e-12))
check("[B-def] the line is fixed: computing the validation error changed no "
"coefficient", np.allclose(line, np.polyfit(tmin[train], tmax[train], 1)))
check("[B-def] no validation day is a training day", not set(val) & set(train))
[B-def] the validation error of a fitted line
40 training days, 20 validation days, 306 held back
the line is h(x) = 10.47 + 1.05 x, the least-squares fit to the 40 training days
[ok] [B-def] the line is the ERM solution: polyfit matches the closed-form least-squares solution
validation error 18.59 = average of 20 squared errors (smallest 0.09, largest 87.11)
[ok] [B-def] the validation error is the average of the per-day losses
[ok] [B-def] the line is fixed: computing the validation error changed no coefficient
[ok] [B-def] no validation day is a training day
The validation error against the risk, for degrees 1, 5 and 9: the training error is below the risk at every degree, and the gap grows with the degree, while for the line it is smaller than the fluctuation of a twenty-day average. The risk here is the average loss over the 306 held-back days, and the block prints how tightly those 306 days pin it.
print("\n[B-risk] the two errors against the risk")
three = []
for degree in (1, 5, 9):
coef = np.polyfit(tmin[train], tmax[train], degree)
three.append((degree, avg_sqerr(tmin[train], tmax[train], coef),
avg_sqerr(tmin[val], tmax[val], coef),
avg_sqerr(tmin[held], tmax[held], coef)))
print(f" degree {degree}: training error {three[-1][1]:6.2f}, validation "
f"error {three[-1][2]:6.2f}, risk {three[-1][3]:8.2f}")
per_day = (tmax[held] - np.polyval(line, tmin[held])) ** 2
noise = float(per_day.std()) / np.sqrt(len(val))
# the risk itself is an expectation and is not computable from a finite
# record; every "risk" printed here is the average loss over the held-back
# days, which is an average of that many numbers and carries its own spread
held_noise = float(per_day.std()) / np.sqrt(len(held))
print(f" the risk column is the average loss over the {len(held)} "
f"held-back days, {three[0][3]:.2f} for the line, itself uncertain "
f"by about {held_noise:.2f}")
print(f" for the line the training error is below the risk by "
f"{three[0][3] - three[0][1]:.2f}, while a {len(val)}-day average "
f"fluctuates by about {noise:.2f}")
check("[B-risk] the held-back average pins the risk more tightly than a "
"validation set of twenty days, by the square root of the ratio of "
"the two sizes",
abs(held_noise - noise * np.sqrt(len(val) / len(held)))
< 0.01 * held_noise)
check("[B-risk] the training error is below the risk at every degree",
all(t < r for _, t, _, r in three))
check("[B-risk] the gap between the training error and the risk grows with "
"the degree",
all(a[3] - a[1] < b[3] - b[1] for a, b in zip(three, three[1:])))
check("[B-risk] for the line that gap is smaller than the fluctuation of a "
"twenty-day average", three[0][3] - three[0][1] < noise)
[B-risk] the two errors against the risk
degree 1: training error 18.96, validation error 18.59, risk 19.41
degree 5: training error 17.93, validation error 20.84, risk 21.45
degree 9: training error 13.80, validation error 35.12, risk 228.96
the risk column is the average loss over the 306 held-back days, 19.41 for the line, itself uncertain by about 1.37
for the line the training error is below the risk by 0.45, while a 20-day average fluctuates by about 5.37
[ok] [B-risk] the held-back average pins the risk more tightly than a validation set of twenty days, by the square root of the ratio of the two sizes
[ok] [B-risk] the training error is below the risk at every degree
[ok] [B-risk] the gap between the training error and the risk grows with the degree
[ok] [B-risk] for the line that gap is smaller than the fluctuation of a twenty-day average
The validation error recomputed over 300 draws of a validation set of each size: the spread shrinks as the set grows. Writes the CSV behind the entry's second figure.
print("\n[B-spread] the fluctuation of the validation error")
SIZES = [5, 10, 20, 40, 80]
DRAWS = 300
risk = float(per_day.mean())
gen = np.random.default_rng(0)
rows = []
for size in SIZES:
means = [float(per_day[gen.choice(len(per_day), size, replace=False)].mean())
for _ in range(DRAWS)]
rows.append((size, float(np.mean(means)), float(np.std(means))))
# the days are drawn without replacement from a finite pool, so the
# spread of the average carries the finite-population correction
expected = (per_day.std() / np.sqrt(size)
* np.sqrt((len(per_day) - size) / (len(per_day) - 1)))
print(f" {size:>2} validation days: mean {rows[-1][1]:6.2f}, spread "
f"{rows[-1][2]:5.2f} (a single day's loss has spread "
f"{per_day.std():.2f}, so {expected:5.2f} is expected)")
check("[B-spread] the spread shrinks as the validation set grows",
all(a[2] > b[2] for a, b in zip(rows, rows[1:])))
check("[B-spread] each mean stays within 10% of the risk",
all(abs(m - risk) < 0.10 * risk for _, m, _ in rows))
def expected_spread(n):
"""The spread of an average of n of the held-back losses, drawn without
replacement: one day's spread over the square root of n, times the
finite-population correction."""
return (per_day.std() / np.sqrt(n)
* np.sqrt((len(per_day) - n) / (len(per_day) - 1)))
check("[B-spread] each spread is within 10% of the standard deviation of a "
"single day's loss divided by the square root of the size, corrected "
"for drawing without replacement",
all(abs(s - expected_spread(n)) < 0.10 * expected_spread(n)
for n, _, s in rows))
with open(OUT_DIR / "valerr_spread.csv", "w") as fh:
fh.write("size,mean,spread\n")
for n, m, s in rows:
fh.write(f"{n},{m:.4f},{s:.4f}\n")
with open(OUT_DIR / "valerr_risk.csv", "w") as fh:
fh.write("size,risk\n")
fh.write(f"{SIZES[0]},{risk:.4f}\n")
fh.write(f"{SIZES[-1]},{risk:.4f}\n")
[B-spread] the fluctuation of the validation error
5 validation days: mean 19.75, spread 11.36 (a single day's loss has spread 24.00, so 10.66 is expected)
10 validation days: mean 19.73, spread 7.03 (a single day's loss has spread 24.00, so 7.48 is expected)
20 validation days: mean 19.38, spread 4.96 (a single day's loss has spread 24.00, so 5.20 is expected)
40 validation days: mean 19.35, spread 3.45 (a single day's loss has spread 24.00, so 3.54 is expected)
80 validation days: mean 19.67, spread 2.26 (a single day's loss has spread 24.00, so 2.31 is expected)
[ok] [B-spread] the spread shrinks as the validation set grows
[ok] [B-spread] each mean stays within 10% of the risk
[ok] [B-spread] each spread is within 10% of the standard deviation of a single day's loss divided by the square root of the size, corrected for drawing without replacement
The learned hypothesis is itself random: over 300 draws of a training set of forty days the ERM quadratic changes every time. The validation error on twenty days drawn independently of the training set still deviates from the risk of that draw's hypothesis by zero on average, while a validation set that shares ten days with the training set, and the training error itself, sit below it.
print("\n[B-cond] the validation error of a hypothesis that is itself random")
# the 366 days stand in for the probability distribution of a day; a draw
# consists of a random training set of forty days, the ERM quadratic
# learned from it, and twenty validation days drawn from the
# rest of the year (so independent of the training set) -- beside a leaky
# validation set in which ten of the twenty days are training days
DEGREE_C, DRAWS_C = 2, 300
gen_c = np.random.default_rng(3)
dev = {"independent validation set": [], "validation set sharing ten "
"training days": [], "training set": []}
for _ in range(DRAWS_C):
perm = gen_c.permutation(len(tmin))
tr, fresh = perm[:40], perm[40:]
coef = np.polyfit(tmin[tr], tmax[tr], DEGREE_C) # changes with every draw
risk_h = avg_sqerr(tmin, tmax, coef) # risk of this draw's hypothesis
val_ind = fresh[:20]
val_leaky = np.concatenate([tr[:10], fresh[20:30]])
dev["independent validation set"].append(
avg_sqerr(tmin[val_ind], tmax[val_ind], coef) - risk_h)
dev["validation set sharing ten training days"].append(
avg_sqerr(tmin[val_leaky], tmax[val_leaky], coef) - risk_h)
dev["training set"].append(avg_sqerr(tmin[tr], tmax[tr], coef) - risk_h)
crows = []
for name, d in dev.items():
d = np.array(d)
crows.append((name, float(d.mean()), float(d.std()),
float(d.std() / np.sqrt(DRAWS_C))))
print(f" {name:>42}: mean deviation {d.mean():+6.2f}, spread "
f"{d.std():5.2f}, standard error of the mean {crows[-1][3]:.2f}")
check("[B-cond] on the independent validation set the deviation from the "
"risk of the learned hypothesis averages to zero: within three "
"standard errors", abs(crows[0][1]) < 3.0 * crows[0][3])
check("[B-cond] the leaky validation set and the training set both sit below "
"the risk by more than three standard errors",
crows[1][1] < -3.0 * crows[1][3] and crows[2][1] < -3.0 * crows[2][3])
check("[B-cond] the training set is the more optimistic of the two",
crows[2][1] < crows[1][1])
with open(OUT_DIR / "valerr_cond.csv", "w") as fh:
fh.write("estimate,mean,spread\n")
for name, m, sd, _ in crows:
fh.write(f"{name},{m:.4f},{sd:.4f}\n")
# ---- preview (checking only)
fig, axes = plt.subplots(1, 3, figsize=(16, 4.0))
ax = axes[0]
ax.plot(tmin5, tmax5, ".", color="0.7", ms=3, label="days of 2020 to 2024")
ax.plot(xs_poly, np.polyval(coef2, xs_poly), "k--", lw=1.4,
label="h2, the interpolating parabola")
for x, y in zip(xv, yv):
ax.plot([x, x], [float(np.polyval(coef2, x)), y], "--", color="red", lw=1.0)
ax.plot(x3, y3, "o", color="black", ms=8, label="training set (3 days)")
ax.plot(xv, yv, "^", mfc="none", mec="red", ms=9, mew=1.5,
label="validation set (3 days)")
ax.set_ylim(tmax5.min() - 12.0, tmax5.max() + 4.0)
ax.set_xlabel("morning minimum temperature (feature, °C)")
ax.set_ylabel("maximum temperature (label, °C)")
ax.set_title(f"Five years at Krems: training error {open_trainerr:.0f}, "
f"validation error {open_valerr:.2f}, risk {open_risk:.2f}",
fontsize=9)
ax.legend(frameon=False, fontsize=8, loc="upper left")
ax = axes[1]
ax.errorbar([r[0] for r in rows], [r[1] for r in rows],
yerr=[r[2] for r in rows], fmt="o", ms=5, color="black",
capsize=3, label="validation error (mean and spread)")
ax.axhline(risk, ls="--", color="0.4", label="risk of the fixed line")
ax.set_xscale("log")
ax.set_xticks(SIZES)
ax.set_xticklabels([str(s) for s in SIZES])
ax.set_xlabel("number of validation days")
ax.set_ylabel("validation error")
ax.set_title("The average fluctuates less as the validation set grows",
fontsize=9)
ax.legend(frameon=False, fontsize=8)
ax = axes[2]
ax.errorbar(range(3), [r[1] for r in crows], yerr=[r[2] for r in crows],
fmt="o", ms=5, color="black", capsize=3,
label="mean and spread over 300 draws")
ax.axhline(0.0, ls="--", color="0.4", label="risk of the learned hypothesis")
ax.set_xticks(range(3))
ax.set_xticklabels(["independent\nvalidation set", "validation set\nsharing "
"10 training days", "training set"], fontsize=8)
ax.set_xlabel("set the average loss is taken over")
ax.set_ylabel("average loss minus risk of the learned hypothesis")
ax.set_title("Quadratic learned anew per draw: only the independent "
"validation set is unbiased", fontsize=9)
ax.legend(frameon=False, fontsize=8)
fig.tight_layout()
fig.savefig(OUT_DIR / "valerr.png", dpi=150)
if INTERACTIVE:
plt.show()
else:
plt.close(fig)
passed = sum(1 for _, ok in report if ok)
print(f"\n{passed}/{len(report)} checks pass")
[B-cond] the validation error of a hypothesis that is itself random
independent validation set: mean deviation +0.13, spread 5.93, standard error of the mean 0.34
validation set sharing ten training days: mean deviation -1.18, spread 5.40, standard error of the mean 0.31
training set: mean deviation -2.29, spread 3.86, standard error of the mean 0.22
[ok] [B-cond] on the independent validation set the deviation from the risk of the learned hypothesis averages to zero: within three standard errors
[ok] [B-cond] the leaky validation set and the training set both sit below the risk by more than three standard errors
[ok] [B-cond] the training set is the more optimistic of the two
23/23 checks pass
B-cond writes when the script runs