Dictionary of Applied Machine Learning · $k$-fold cross-validation ($k$-fold CV)
Updated on 2026-11-03
Numerical companion to the entry $k$-fold cross-validation ($k$-fold CV): it recomputes what the entry states and prints one line per check
Carries out the entry's four-day weather example and backs its claims: a single split can hand back a small validation error for a hypothesis that is useless away from the data, and averaging over all k splits exposes that; and the choice of k trades bias against variance of the estimate. Self-contained (numpy and matplotlib only), fixed seed.
Run it without installing anything:uv run https://dictionaryofml.org/terms/kfoldcv.py
uv downloads this script and the pinned NumPy and Matplotlib it needs, then runs it; the script fetches any input file it uses. To keep the output files, download kfoldcv.py into a folder and run uv run kfoldcv.py there. With NumPy and Matplotlib already installed, python3 kfoldcv.py, from any directory — it writes its output files into the current directory. Fixed seeds, so the printed numbers reproduce exactly. Download kfoldcv.py · Notebook · Open in Colab
Before running a downloaded copy, scan it: uvx bandit -q -t B102,B307,B602,B605 kfoldcv.py (B102,B307,B602,B605 are the exec, eval and shell checks of bandit) reports nothing: the script runs no shell command and evaluates no generated code.
One cell per block of the script: the code, and what that code printed when it last ran here
"""
kfoldcv.py — numerical companion to the glossary entry 'k-fold
cross-validation (k-fold CV)'.
Purpose
-------
Carries out the entry's four-day weather example and backs its claims: a
single split can hand back a small validation error for a hypothesis that
is useless away from the data, and averaging over all k splits exposes
that; and the choice of k trades bias against variance of the estimate.
Self-contained (numpy and matplotlib only), fixed seed.
Setup
-----
Four days at Krems an der Donau, feature x = morning minimum temperature
and label y = maximum daytime temperature, both in degrees C. The method
under study is polynomial regression of degree three with the squared
error loss. A cubic has four coefficients and a training set here holds
three days, so the system is underdetermined: the minimum-norm
least-squares solution is used, and it passes exactly through its three
training days. Each day serves once as the validation set, so k = 4.
Blocks
------
[B-days] The four days, and the cubic learned from all four. With four
points and four coefficients that fit is exact, so the training
error is zero and says nothing about the risk — which is what
makes a held-out day necessary.
[B-split] One split, the flattering one: hold out the first day. The
cubic through the other three has zero training error and a
validation error of 1.37, so it looks like a good hypothesis.
Over the plotted range it runs from 5.5 to 33.9 degrees while
the four labels lie between 15 and 23, and it departs by 13.0
degrees from a straight line through all four days: it is
useless away from its three days, and the single split hid
that.
[B-cv] All four splits: validation errors 1.37, 8.05, 7.02 and 72.42,
averaging to the 4-fold CV estimate 22.22. The flattering
split understates it sixteenfold.
[B-k] The choice of k, in a SEPARATE setting from the four days
above: a linear model rather than a cubic, and 2000 datasets
of fifteen days drawn INDEPENDENTLY from y = 4 + 0.8 x plus
noise (not resampled from one dataset — independence is what
lets this estimate a variance that cannot be read off a
single dataset). For k = 2, 3, 5 and 15 (leave-one-out) the
estimate's bias with respect to the expected loss falls with
k, since each training set holds a fraction (k-1)/k of the
days, and so does its spread across datasets.
Outputs
-------
kfoldcv_days.csv : x, y of the four days.
kfoldcv_lines.csv : x, full, line1..line4 — the cubic on all four days
and the cubic of each of the four splits.
kfoldcv_panels.csv : x, train1, val1, ..., train4, val4 — the days by set
membership per split, non-members written as nan for
'unbounded coords=discard'.
kfoldcv_folds.csv : fold index and its validation error.
kfoldcv_k.csv : k, bias of the k-fold CV estimate, its spread.
kfoldcv.png : matplotlib preview (checking only).
"""
# /// script
# requires-python = ">=3.10"
# dependencies = ["matplotlib==3.10.7", "numpy==2.3.4"]
# ///
import numpy as np
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
from pathlib import Path
OUT_DIR = Path(__file__).parent
report = []
def check(name, ok):
report.append((name, bool(ok)))
print(f" [{'ok' if ok else 'FAIL'}] {name}")
DEGREE = 3
XLO, XHI = 0.0, 11.0 # the plotted range of x
def cubic(xs, ys):
"""Minimum-norm least-squares polynomial of degree DEGREE. With three
days and four coefficients the system is underdetermined, so a plain
solve is not defined; lstsq picks the smallest-norm solution, which
still passes exactly through the training days."""
coef, *_ = np.linalg.lstsq(np.vander(xs, DEGREE + 1), ys, rcond=None)
return coef
def sqerr(xs, ys, coef):
return float(np.mean((np.polyval(coef, xs) - ys) ** 2))
4 days: morning minimum 1, 6, 8, 10; maximum daytime 15, 18, 15, 23
training error of the cubic on all four days: 2.07e-26
The four days, and the cubic learned from all four. With four points and four coefficients that fit is exact, so the training error is zero and says nothing about the risk — which is what makes a held-out day necessary.
x = np.array([1.0, 6.0, 8.0, 10.0])
y = np.array([15.0, 18.0, 15.0, 23.0])
m = len(x)
coef_full = cubic(x, y)
grid = np.linspace(XLO, XHI, 400)
print(f" {m} days: morning minimum "
+ ", ".join(f"{v:.0f}" for v in x)
+ "; maximum daytime " + ", ".join(f"{v:.0f}" for v in y))
print(f" training error of the cubic on all four days: "
f"{sqerr(x, y, coef_full):.2e}")
check("[B-days] the cubic on all four days has (almost) zero training error",
sqerr(x, y, coef_full) < 1e-16)
[ok] [B-days] the cubic on all four days has (almost) zero training error
holding out day 1 (x = 1): validation error 1.37, curve on [0, 11] runs 5.5..33.9
One split, the flattering one: hold out the first day. The cubic through the other three has zero training error and a validation error of 1.37, so it looks like a good hypothesis. Over the plotted range it runs from 5.5 to 33.9 degrees while the four labels lie between 15 and 23, and it departs by 13.0 degrees from a straight line through all four days: it is useless away from its three days, and the single split hid that.
# line_all is the reference a reader would accept as sane; the cubic of this
# split is judged by how far it wanders from it, not by its validation error
HELD = 0 # index of the held-out day
tr = np.array([i for i in range(m) if i != HELD])
coef_one = cubic(x[tr], y[tr])
err_one = (np.polyval(coef_one, x[HELD]) - y[HELD]) ** 2
curve_one = np.polyval(coef_one, grid)
print(f" holding out day {HELD + 1} (x = {x[HELD]:.0f}): validation error "
f"{err_one:.2f}, curve on [{XLO:.0f}, {XHI:.0f}] runs "
f"{curve_one.min():.1f}..{curve_one.max():.1f}")
check("[B-split] the split's training error is zero: the cubic interpolates "
"its three days",
float(np.abs(np.polyval(coef_one, x[tr]) - y[tr]).max()) < 1e-8)
check("[B-split] its validation error is small, under 3",
err_one < 3.0)
line_all = np.polyfit(x, y, 1)
gap = float(np.abs(curve_one - np.polyval(line_all, grid)).max())
print(f" away from its three days it departs from a straight line that "
f"fits all four by up to {gap:.1f} degrees")
check("[B-split] yet the hypothesis is useless away from its three days: it "
"leaves the range of the observed labels on both sides",
curve_one.min() < y.min() - 1.0 and curve_one.max() > y.max() + 1.0)
check("[B-split] and it departs from a line through all four days by more "
"than ten degrees",
gap > 10.0)
[ok] [B-split] the split's training error is zero: the cubic interpolates its three days
[ok] [B-split] its validation error is small, under 3
away from its three days it departs from a straight line that fits all four by up to 13.0 degrees
[ok] [B-split] yet the hypothesis is useless away from its three days: it leaves the range of the observed labels on both sides
[ok] [B-split] and it departs from a line through all four days by more than ten degrees
validation errors 1.37, 8.05, 7.02, 72.42; 4-fold CV estimate 22.22
All four splits: validation errors 1.37, 8.05, 7.02 and 72.42, averaging to the 4-fold CV estimate 22.22. The flattering split understates it sixteenfold.
errs, coefs = [], []
for b in range(m):
keep = np.array([i for i in range(m) if i != b])
c = cubic(x[keep], y[keep])
coefs.append(c)
errs.append(float((np.polyval(c, x[b]) - y[b]) ** 2))
cv = float(np.mean(errs))
print(" validation errors " + ", ".join(f"{e:.2f}" for e in errs)
+ f"; 4-fold CV estimate {cv:.2f}")
check("[B-cv] every split interpolates its own three days",
all(float(np.abs(np.polyval(coefs[b],
x[[i for i in range(m) if i != b]])
- y[[i for i in range(m) if i != b]]).max()) < 1e-8
for b in range(m)))
check(f"[B-cv] the flattering split understates the CV estimate "
f"({cv:.2f}) more than tenfold",
cv > 10 * err_one)
[ok] [B-cv] every split interpolates its own three days [ok] [B-cv] the flattering split understates the CV estimate (22.22) more than tenfold
The choice of k, in a SEPARATE setting from the four days above: a linear model rather than a cubic, and 2000 datasets of fifteen days drawn INDEPENDENTLY from y = 4 + 0.8 x plus noise (not resampled from one dataset — independence is what lets this estimate a variance that cannot be read off a single dataset). For k = 2, 3, 5 and 15 (leave-one-out) the estimate's bias with respect to the expected loss falls with k, since each training set holds a fraction (k-1)/k of the days, and so does its spread across datasets.
rng = np.random.default_rng(0)
R, N, NOISE = 2000, 15, 1.5
ks = [2, 3, 5, 15]
def erm_line(xs, ys):
return np.polyfit(xs, ys, 1)
def draw(n):
xs = rng.uniform(-15.0, 5.0, n)
return xs, 4.0 + 0.8 * xs + rng.normal(0.0, NOISE, n)
x_fresh, y_fresh = draw(20000) # for the expected loss
est = {k: [] for k in ks}
true_risk = []
for _ in range(R):
xs, ys = draw(N)
true_risk.append(sqerr(x_fresh, y_fresh, erm_line(xs, ys)))
for k in ks:
f = np.arange(N) % k
e = [sqerr(xs[f == b], ys[f == b], erm_line(xs[f != b], ys[f != b]))
for b in range(k)]
est[k].append(float(np.mean(e)))
risk = float(np.mean(true_risk))
rows = [(k, float(np.mean(est[k]) - risk), float(np.std(est[k]))) for k in ks]
bias = {r[0]: r[1] for r in rows}
check(f"[B-k] expected loss of ERM on 15 days {risk:.2f}; bias of the "
f"k-fold CV estimate "
+ ", ".join(f"k={r[0]}: {r[1]:+.2f}" for r in rows),
bias[2] > bias[5] > bias[15] and bias[2] > 0.1 and abs(bias[15]) < 0.1)
# ---- CSV ------------------------------------------------------------------
with open(OUT_DIR / "kfoldcv_days.csv", "w") as fh:
fh.write("x,y\n")
for a, b_ in zip(x, y):
fh.write(f"{a:.4f},{b_:.4f}\n")
with open(OUT_DIR / "kfoldcv_lines.csv", "w") as fh:
fh.write("x,full," + ",".join(f"line{b + 1}" for b in range(m)) + "\n")
for i, gx in enumerate(grid):
vals = [np.polyval(coef_full, gx)] + [np.polyval(c, gx) for c in coefs]
fh.write(f"{gx:.4f}," + ",".join(f"{v:.4f}" for v in vals) + "\n")
with open(OUT_DIR / "kfoldcv_panels.csv", "w") as fh:
cols = []
for b in range(m):
cols += [f"train{b + 1}", f"val{b + 1}"]
fh.write("x," + ",".join(cols) + "\n")
for i in range(m):
row = [f"{x[i]:.4f}"]
for b in range(m):
row.append("nan" if i == b else f"{y[i]:.4f}") # train
row.append(f"{y[i]:.4f}" if i == b else "nan") # val
fh.write(",".join(row) + "\n")
with open(OUT_DIR / "kfoldcv_folds.csv", "w") as fh:
fh.write("fold,valerr\n")
for b, e in enumerate(errs):
fh.write(f"{b + 1},{e:.4f}\n")
with open(OUT_DIR / "kfoldcv_k.csv", "w") as fh:
fh.write("k,bias,spread\n")
for k, bi, sd in rows:
fh.write(f"{k},{bi:.4f},{sd:.4f}\n")
[ok] [B-k] expected loss of ERM on 15 days 2.62; bias of the k-fold CV estimate k=2: +0.48, k=3: +0.20, k=5: +0.08, k=15: +0.01 8/8 checks pass wrote kfoldcv_days.csv, kfoldcv_lines.csv, kfoldcv_panels.csv, kfoldcv_folds.csv, kfoldcv_k.csv, kfoldcv.png in /Users/junga1/dictionaryappliedml/pythondemos
fig = plt.figure(figsize=(10.0, 5.6))
ax = fig.add_subplot(2, 1, 1)
ax.plot(grid, np.polyval(coef_full, grid), "-", color="black", lw=1.4,
label="learned hypothesis")
ax.plot(x, y, "o", color="black", ms=7, label="data points")
ax.set_xlim(XLO, XHI)
ax.set_xlabel("morning minimum temperature $x$")
ax.set_ylabel("maximum daytime\ntemperature $y$")
ax.set_title("four days and the degree-3 hypothesis learned from all of them",
fontsize=9)
ax.legend(frameon=False, fontsize=8)
for b in range(m):
axb = fig.add_subplot(2, m, m + b + 1)
keep = [i for i in range(m) if i != b]
axb.plot(grid, np.polyval(coefs[b], grid), "-", color="black", lw=1.2)
axb.plot(x[keep], y[keep], "o", color="black", ms=6)
axb.plot(x[b], y[b], "s", color="red", mfc="none", ms=10, mew=2.0)
axb.set_xlim(XLO, XHI)
axb.set_ylim(min(0, min(np.polyval(c, grid).min() for c in coefs)) - 2,
max(np.polyval(c, grid).max() for c in coefs) + 2)
axb.set_title(f"$b={b + 1}$, val. error {errs[b]:.2f}", fontsize=8)
axb.set_xlabel("$x$", fontsize=8)
if b == 0:
axb.set_ylabel("$y$", fontsize=8)
axb.tick_params(labelsize=7)
fig.tight_layout()
fig.savefig(OUT_DIR / "kfoldcv.png", dpi=110)
n_ok = sum(ok for _, ok in report)
print(f"\n{n_ok}/{len(report)} checks pass")
print(f"wrote kfoldcv_days.csv, kfoldcv_lines.csv, kfoldcv_panels.csv, "
f"kfoldcv_folds.csv, kfoldcv_k.csv, kfoldcv.png in {OUT_DIR}")
if n_ok != len(report):
raise SystemExit(1)

B-preview writes when the script runs