B-kfold writes when the script runsDictionary of Applied Machine Learning · validation set
Numerical companion to the entry validation set: it recomputes what the entry states and prints one line per check
The weather narrative of the 'training set' entry, continued: the same three training days and the same two hypotheses (straight line, degree-two polynomial), plus three validation days held back from model training. The validation error ranks the two hypotheses opposite to the training error, the hypothesis with the smaller validation error is selected (model selection), and three further test days assess the selected hypothesis. The demo also evaluates the Hoeffding lower bound on the validation-set size behind the entry's third figure. Self-contained (numpy/matplotlib only), deterministic.
Run it without installing anything:uv run https://dictionaryofml.org/terms/valset.py
uv downloads this script and the pinned NumPy and Matplotlib it needs, then runs it; the script fetches any input file it uses. To keep the output files, download valset.py into a folder and run uv run valset.py there. With NumPy and Matplotlib already installed, python3 valset.py, from any directory — it writes its output files into the current directory. Fixed seeds, so the printed numbers reproduce exactly. Download valset.py · Notebook · Open in Colab
One cell per block of the script: the code, and what that code printed when it last ran here
"""
valset.py -- numerical companion to the entry 'validation set'.
The weather narrative of the 'training set' entry, continued: the same
three training days and the same two hypotheses (straight line,
degree-two polynomial), plus three validation days held back from model
training. The validation error ranks the two hypotheses opposite to
the training error, the hypothesis with the smaller validation error is
selected (model selection), and three further test days assess the
selected hypothesis. The demo also evaluates the Hoeffding lower bound
on the validation-set size behind the entry's third figure.
Self-contained (numpy/matplotlib only), deterministic.
Blocks
------
[B-data] Recreate the three training days of the 'training set'
entry and learn the two hypotheses from them; check that
the fitted coefficients match those of trainset.py, so the
figures of the two entries show the same curves.
[B-valset] Three validation days held back from model training. The
validation error of the polynomial far exceeds that of the
line, reversing the training-error ranking.
[B-modelsel] Select the hypothesis with the smaller validation error
(the line); evaluate the selected hypothesis on three test
days that entered neither training nor selection.
[B-bound] The Hoeffding lower bound ln(2/delta) / (2 epsilon^2) on the
validation-set size: check the entry's example (185 data
points for epsilon = 0.1, delta = 0.05) and print the bound
for the three curves of the entry's figure.
[B-verify] A simulation check of that bound: for each pair (epsilon,
delta) of the entry's third figure, 200,000 validation
sets of exactly the required size are drawn for a loss
with values in {0, 1} and risk 1/2 (the largest variance a
loss with values in [0,1] can have), and the share of
them whose validation
error deviates from the risk by more than epsilon is
measured. Every share stays below its delta, and below
delta/4, so the bound holds and is conservative; a
validation set eight times smaller violates it. Written
to valset_dev_d001.csv, _d01.csv and _d05.csv for the
entry's fourth figure (author, 2026-10-17).
[B-kfold] 3-fold cross-validation on the six pooled days: each fold
yields a noisy validation error and their average is the CV
estimate. Over 300 replicate datasets, the CV estimate
varies far less than a single split with folds of the same
size, backing the entry's remedy for scarce data points.
Outputs
-------
pythondemos/valset.png : preview figure (checking only).
pythondemos/valset_train.csv : the three training days (x = morning
minimum temperature, y = maximum
daytime temperature)
pythondemos/valset_val.csv : the three validation days
pythondemos/valset_test.csv : the three test days
pythondemos/valset_line.csv : the fitted line on a two-point grid
pythondemos/valset_poly.csv : the fitted polynomial on a dense grid
pythondemos/valset_dev_d001.csv : eps,freq -- measured deviation shares
pythondemos/valset_dev_d01.csv for delta = 0.01, 0.1 and 0.5
pythondemos/valset_dev_d05.csv
"""
import numpy as np
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
from pathlib import Path
OUT_DIR = Path(__file__).parent
report = []
def check(name, ok):
report.append((name, bool(ok)))
print(f" [{'ok' if ok else 'FAIL'}] {name}")
def days(n, gen, lo=-15.0, hi=5.0, w1=0.8, w0=4.0, noise=2.0):
"""n days: morning minimum x and maximum daytime temperature y."""
x = np.sort(gen.uniform(lo, hi, n))
y = w0 + w1 * x + gen.normal(0.0, noise, n)
return x, y
def avg_sqerr(x, y, coeffs):
return float(np.mean((y - np.polyval(coeffs, x)) ** 2))
Recreate the three training days of the 'training set' entry and learn the two hypotheses from them; check that the fitted coefficients match those of trainset.py, so the figures of the two entries show the same curves.
print("[B-data] the three training days and the two hypotheses of the "
"'training set' entry")
x, y = days(3, np.random.default_rng(100))
c_line = np.polyfit(x, y, 1)
c_poly = np.polyfit(x, y, 2)
trainerr_line = avg_sqerr(x, y, c_line)
trainerr_poly = avg_sqerr(x, y, c_poly)
print(f" fitted line: max temperature = {c_line[1]:.2f} + {c_line[0]:.2f}"
f" * min temperature; training error {trainerr_line:.3f}")
check("the fitted line matches trainset.py (same coefficients)",
abs(c_line[0] - 0.8688) < 5e-3 and abs(c_line[1] - 4.6785) < 5e-3)
check("the polynomial's training error is zero (up to round-off)",
trainerr_poly < 1e-10)
[B-data] the three training days and the two hypotheses of the 'training set' entry
fitted line: max temperature = 4.68 + 0.87 * min temperature; training error 2.872
[ok] the fitted line matches trainset.py (same coefficients)
[ok] the polynomial's training error is zero (up to round-off)
Three validation days held back from model training. The validation error of the polynomial far exceeds that of the line, reversing the training-error ranking.
print("\n[B-valset] three validation days held back from model training")
xv, yv = days(3, np.random.default_rng(1100))
for xr, yr in zip(xv, yv):
print(f" morning minimum {xr:6.1f} C, maximum daytime {yr:6.1f} C")
valerr_line = avg_sqerr(xv, yv, c_line)
valerr_poly = avg_sqerr(xv, yv, c_poly)
print(f" validation error: line {valerr_line:.2f}, "
f"polynomial {valerr_poly:.2f}")
check("the validation days are disjoint from the training days",
len(set(np.round(xv, 6)) & set(np.round(x, 6))) == 0)
check("the validation error reverses the training-error ranking",
trainerr_poly < trainerr_line and valerr_poly > valerr_line)
check("the polynomial's validation error far exceeds the line's",
valerr_poly > 3 * valerr_line)
[B-valset] three validation days held back from model training
morning minimum -11.1 C, maximum daytime -4.8 C
morning minimum -10.5 C, maximum daytime -7.5 C
morning minimum -0.4 C, maximum daytime 4.7 C
validation error: line 3.02, polynomial 17.29
[ok] the validation days are disjoint from the training days
[ok] the validation error reverses the training-error ranking
[ok] the polynomial's validation error far exceeds the line's
Select the hypothesis with the smaller validation error (the line); evaluate the selected hypothesis on three test days that entered neither training nor selection.
print("\n[B-modelsel] the validation error selects the line; test days "
"assess it")
selected = "line" if valerr_line < valerr_poly else "polynomial"
xt, yt = days(3, np.random.default_rng(3100))
testerr_line = avg_sqerr(xt, yt, c_line)
print(f" selected hypothesis: {selected}; "
f"test error of the line {testerr_line:.2f}")
check("the smaller validation error selects the line", selected == "line")
check("the test days entered neither training nor selection",
len(set(np.round(xt, 6)) & set(np.round(np.r_[x, xv], 6))) == 0)
[B-modelsel] the validation error selects the line; test days assess it
selected hypothesis: line; test error of the line 1.34
[ok] the smaller validation error selects the line
[ok] the test days entered neither training nor selection
The Hoeffding lower bound ln(2/delta) / (2 epsilon^2) on the validation-set size: check the entry's example (185 data points for epsilon = 0.1, delta = 0.05) and print the bound for the three curves of the entry's figure.
print("\n[B-bound] the Hoeffding lower bound on the validation-set size")
def bound(delta, band):
return np.log(2.0 / delta) / (2.0 * band ** 2)
print(f" epsilon = 0.1, delta = 0.05: {bound(0.05, 0.1):.1f} "
f"-> 185 data points")
for delta in (0.01, 0.1, 0.5):
print(f" delta = {delta}: bound at epsilon = 0.1 is "
f"{bound(delta, 0.1):.0f}")
check("the entry's example holds: 185 data points suffice",
int(np.ceil(bound(0.05, 0.1))) == 185)
check("the bound grows as the band narrows (0.05 needs 4x more than 0.1)",
abs(bound(0.05, 0.05) / bound(0.05, 0.1) - 4.0) < 1e-12)
[B-bound] the Hoeffding lower bound on the validation-set size
epsilon = 0.1, delta = 0.05: 184.4 -> 185 data points
delta = 0.01: bound at epsilon = 0.1 is 265
delta = 0.1: bound at epsilon = 0.1 is 150
delta = 0.5: bound at epsilon = 0.1 is 69
[ok] the entry's example holds: 185 data points suffice
[ok] the bound grows as the band narrows (0.05 needs 4x more than 0.1)
A simulation check of that bound: for each pair (epsilon, delta) of the entry's third figure, 200,000 validation sets of exactly the required size are drawn for a loss with values in {0, 1} and risk 1/2 (the largest variance a loss with values in [0,1] can have), and the share of them whose validation error deviates from the risk by more than epsilon is measured. Every share stays below its delta, and below delta/4, so the bound holds and is conservative; a validation set eight times smaller violates it. Written to valset_dev_d001.csv, _d01.csv and _d05.csv for the entry's fourth figure (author, 2026-10-17).
print("\n[B-verify] how often the validation error deviates by more than "
"epsilon")
# A loss with values in {0, 1} and risk 1/2: the validation error over a
# validation set of size m is then Binomial(m, 1/2) / m, so a replicate
# validation set is one binomial draw. Risk 1/2 maximizes the variance of a
# loss with values in [0, 1], which is the hardest case for the bound.
RISK = 0.5
N_REP = 200_000
BANDS = (0.03, 0.04, 0.05, 0.07, 0.1, 0.14, 0.2)
DELTAS = ((0.01, "d001"), (0.1, "d01"), (0.5, "d05"))
gen_dev = np.random.default_rng(11)
def deviation_share(size, band, n_rep, gen):
"""Share of n_rep validation sets of the given size whose validation
error deviates from the risk by more than the half-width."""
valerr = gen.binomial(size, RISK, size=n_rep) / size
return float(np.mean(np.abs(valerr - RISK) > band))
shares = {}
for delta, tag in DELTAS:
row = []
for band in BANDS:
size = int(np.ceil(bound(delta, band)))
row.append((band, size, deviation_share(size, band, N_REP, gen_dev)))
shares[delta] = row
print(f" delta = {delta}: " + ", ".join(
f"eps {b} (|valset| {sz}) -> {sh:.4f}" for b, sz, sh in row))
check("[B-verify] every measured share stays below its delta",
all(sh < delta for delta, row in shares.items() for _, _, sh in row))
check("[B-verify] every measured share stays below delta/4 (the bound is "
"conservative)",
all(sh < delta / 4.0 for delta, row in shares.items()
for _, _, sh in row))
small = deviation_share(max(int(np.ceil(bound(0.1, 0.1) / 8)), 1), 0.1,
N_REP, gen_dev)
print(f" a validation set eight times smaller than the bound requires "
f"(delta = 0.1, eps = 0.1): {small:.4f} > 0.1")
check("[B-verify] a validation set eight times smaller violates the bound",
small > 0.1)
[B-verify] how often the validation error deviates by more than epsilon
delta = 0.01: eps 0.03 (|valset| 2944) -> 0.0012, eps 0.04 (|valset| 1656) -> 0.0010, eps 0.05 (|valset| 1060) -> 0.0011, eps 0.07 (|valset| 541) -> 0.0010, eps 0.1 (|valset| 265) -> 0.0009, eps 0.14 (|valset| 136) -> 0.0008, eps 0.2 (|valset| 67) -> 0.0012
delta = 0.1: eps 0.03 (|valset| 1665) -> 0.0142, eps 0.04 (|valset| 937) -> 0.0152, eps 0.05 (|valset| 600) -> 0.0145, eps 0.07 (|valset| 306) -> 0.0143, eps 0.1 (|valset| 150) -> 0.0110, eps 0.14 (|valset| 77) -> 0.0117, eps 0.2 (|valset| 38) -> 0.0137
delta = 0.5: eps 0.03 (|valset| 771) -> 0.0961, eps 0.04 (|valset| 434) -> 0.0939, eps 0.05 (|valset| 278) -> 0.1055, eps 0.07 (|valset| 142) -> 0.1101, eps 0.1 (|valset| 70) -> 0.0722, eps 0.14 (|valset| 36) -> 0.0654, eps 0.2 (|valset| 18) -> 0.0960
[ok] [B-verify] every measured share stays below its delta
[ok] [B-verify] every measured share stays below delta/4 (the bound is conservative)
a validation set eight times smaller than the bound requires (delta = 0.1, eps = 0.1): 0.3587 > 0.1
[ok] [B-verify] a validation set eight times smaller violates the bound
3-fold cross-validation on the six pooled days: each fold yields a noisy validation error and their average is the CV estimate. Over 300 replicate datasets, the CV estimate varies far less than a single split with folds of the same size, backing the entry's remedy for scarce data points.
print("\n[B-kfold] averaging noisy per-fold validation errors is more "
"reliable")
X_all, Y_all = np.r_[x, xv], np.r_[y, yv]
def cv3(xs, ys, gen):
"""Per-fold validation errors of 3-fold CV of the linear model."""
folds = gen.permutation(len(xs)).reshape(3, 2)
errs = []
for hold in folds:
keep = np.ones(len(xs), bool)
keep[hold] = False
c = np.polyfit(xs[keep], ys[keep], 1)
errs.append(avg_sqerr(xs[hold], ys[hold], c))
return errs
fold_errs = cv3(X_all, Y_all, np.random.default_rng(5))
print(f" per-fold validation errors: "
f"{', '.join(f'{v:.2f}' for v in fold_errs)}; "
f"average {np.mean(fold_errs):.2f}")
check("the per-fold validation errors are noisy (spread over a 3x range)",
max(fold_errs) > 3 * min(fold_errs))
gen = np.random.default_rng(9)
singles, cv_avgs = [], []
for _ in range(300):
xr, yr = days(6, gen)
c = np.polyfit(xr[:4], yr[:4], 1)
singles.append(avg_sqerr(xr[4:], yr[4:], c))
cv_avgs.append(float(np.mean(cv3(xr, yr, gen))))
std_single, std_cv = float(np.std(singles)), float(np.std(cv_avgs))
print(f" over 300 replicate datasets: std of a single split "
f"{std_single:.1f}, std of the 3-fold CV average {std_cv:.1f}")
check("the CV average varies far less than a single split",
std_single > 2 * std_cv)
# --------------------------------------------------- outputs: CSVs, preview
xs_line = np.array([-14.0, 5.0])
xs_poly = np.linspace(-14.0, 5.0, 200)
def write_csv(name, cols, header):
np.savetxt(OUT_DIR / name, np.column_stack(cols), delimiter=",",
header=header, comments="", fmt="%.4f")
write_csv("valset_train.csv", (x, y), "x,y")
write_csv("valset_val.csv", (xv, yv), "x,y")
write_csv("valset_test.csv", (xt, yt), "x,y")
write_csv("valset_line.csv", (xs_line, np.polyval(c_line, xs_line)),
"x,yhat")
write_csv("valset_poly.csv", (xs_poly, np.polyval(c_poly, xs_poly)),
"x,yhat")
for delta, tag in DELTAS:
write_csv(f"valset_dev_{tag}.csv",
(np.array([b for b, _, _ in shares[delta]]),
np.array([sh for _, _, sh in shares[delta]])),
"eps,freq")
fig, axes = plt.subplots(1, 3, figsize=(13.8, 3.6))
ax = axes[0]
ax.plot(xs_poly, np.polyval(c_poly, xs_poly), "--", color="black", lw=1.2,
label="polynomial")
ax.plot(xs_line, np.polyval(c_line, xs_line), "-", color="black", lw=1.2,
label="line (selected)")
ax.plot(x, y, "o", ms=6, color="black", label="training set")
ax.plot(xv, yv, "^", ms=7, markerfacecolor="none", markeredgecolor="black",
label="validation set")
ax.plot(xt, yt, "s", ms=6, markerfacecolor="none", markeredgecolor="black",
label="test set")
ax.set_xlabel("morning minimum temperature")
ax.set_ylabel("maximum daytime temperature")
ax.set_title("validation days rank the two hypotheses", fontsize=9)
ax.legend(frameon=False, fontsize=7)
ax = axes[1]
band = np.linspace(0.02, 0.5, 200)
for delta, style in ((0.01, "-"), (0.1, "--"), (0.5, ":")):
ax.semilogy(band, bound(delta, band), style, color="black", lw=1.2,
label=f"confidence level {1 - delta:g}")
ax.set_xlabel("uncertainty band half-width")
ax.set_ylabel("required validation-set size")
ax.set_title("Hoeffding lower bound on the validation-set size", fontsize=9)
ax.legend(frameon=False, fontsize=8)
ax = axes[2]
for (delta, _tag), style in zip(DELTAS, ("-", "--", ":")):
bands = [b for b, _, _ in shares[delta]]
freqs = [sh for _, _, sh in shares[delta]]
ax.semilogy(bands, freqs, style, color="black", lw=1.2, marker="o", ms=3.5,
label=f"measured, $\\delta = {delta:g}$")
# the nominal delta as a short segment at the right margin: a full-width
# line would run along the measured curve of the next larger delta
ax.semilogy([0.207, 0.218], [delta, delta], "-", color="0.6", lw=1.6)
ax.text(0.222, delta, f"$\\delta = {delta:g}$", fontsize=7, color="0.4",
va="center")
ax.set_xlabel("uncertainty band half-width $\\epsilon$")
ax.set_ylabel("share deviating by more than $\\epsilon$")
ax.set_xlim(0.02, 0.262)
ax.set_ylim(5e-4, 2.5)
ax.set_title("the bound holds, and is conservative", fontsize=9)
ax.legend(frameon=False, fontsize=7, loc="upper left")
fig.tight_layout()
fig.savefig(OUT_DIR / "valset.png", dpi=110)
n_ok = sum(ok for _, ok in report)
print(f"\n{n_ok}/{len(report)} checks pass")
print("wrote valset.png")
if n_ok != len(report):
raise SystemExit(1)
[B-kfold] averaging noisy per-fold validation errors is more reliable
per-fold validation errors: 11.21, 2.80, 3.29; average 5.77
[ok] the per-fold validation errors are noisy (spread over a 3x range)
over 300 replicate datasets: std of a single split 55.8, std of the 3-fold CV average 14.4
[ok] the CV average varies far less than a single split
14/14 checks pass
wrote valset.png
B-kfold writes when the script runs