Dictionary of Applied Machine Learning · $k$-fold cross-validation ($k$-fold CV)

$k$-fold cross-validation ($k$-fold CV) — Python demo

Updated on 2026-11-03

Numerical companion to the entry $k$-fold cross-validation ($k$-fold CV): it recomputes what the entry states and prints one line per check

Carries out the entry's four-day weather example and backs its claims: a single split can hand back a small validation error for a hypothesis that is useless away from the data, and averaging over all k splits exposes that; and the choice of k trades bias against variance of the estimate. Self-contained (numpy and matplotlib only), fixed seed.

Run it without installing anything:
uv run https://dictionaryofml.org/terms/kfoldcv.py
uv downloads this script and the pinned NumPy and Matplotlib it needs, then runs it; the script fetches any input file it uses. To keep the output files, download kfoldcv.py into a folder and run uv run kfoldcv.py there. With NumPy and Matplotlib already installed, python3 kfoldcv.py, from any directory — it writes its output files into the current directory. Fixed seeds, so the printed numbers reproduce exactly. Download kfoldcv.py · Notebook · Open in Colab

Before running a downloaded copy, scan it: uvx bandit -q -t B102,B307,B602,B605 kfoldcv.py (B102,B307,B602,B605 are the exec, eval and shell checks of bandit) reports nothing: the script runs no shell command and evaluates no generated code.

The script, block by block

One cell per block of the script: the code, and what that code printed when it last ran here

setup

"""
kfoldcv.py — numerical companion to the glossary entry 'k-fold
cross-validation (k-fold CV)'.

Purpose
-------
Carries out the entry's four-day weather example and backs its claims: a
single split can hand back a small validation error for a hypothesis that
is useless away from the data, and averaging over all k splits exposes
that; and the choice of k trades bias against variance of the estimate.
Self-contained (numpy and matplotlib only), fixed seed.

Setup
-----
Four days at Krems an der Donau, feature x = morning minimum temperature
and label y = maximum daytime temperature, both in degrees C.  The method
under study is polynomial regression of degree three with the squared
error loss.  A cubic has four coefficients and a training set here holds
three days, so the system is underdetermined: the minimum-norm
least-squares solution is used, and it passes exactly through its three
training days.  Each day serves once as the validation set, so k = 4.

Blocks
------
[B-days]   The four days, and the cubic learned from all four.  With four
           points and four coefficients that fit is exact, so the training
           error is zero and says nothing about the risk — which is what
           makes a held-out day necessary.
[B-split]  One split, the flattering one: hold out the first day.  The
           cubic through the other three has zero training error and a
           validation error of 1.37, so it looks like a good hypothesis.
           Over the plotted range it runs from 5.5 to 33.9 degrees while
           the four labels lie between 15 and 23, and it departs by 13.0
           degrees from a straight line through all four days: it is
           useless away from its three days, and the single split hid
           that.
[B-cv]     All four splits: validation errors 1.37, 8.05, 7.02 and 72.42,
           averaging to the 4-fold CV estimate 22.22.  The flattering
           split understates it sixteenfold.
[B-k]      The choice of k, in a SEPARATE setting from the four days
           above: a linear model rather than a cubic, and 2000 datasets
           of fifteen days drawn INDEPENDENTLY from y = 4 + 0.8 x plus
           noise (not resampled from one dataset — independence is what
           lets this estimate a variance that cannot be read off a
           single dataset).  For k = 2, 3, 5 and 15 (leave-one-out) the
           estimate's bias with respect to the expected loss falls with
           k, since each training set holds a fraction (k-1)/k of the
           days, and so does its spread across datasets.

Outputs
-------
kfoldcv_days.csv   : x, y of the four days.
kfoldcv_lines.csv  : x, full, line1..line4 — the cubic on all four days
                     and the cubic of each of the four splits.
kfoldcv_panels.csv : x, train1, val1, ..., train4, val4 — the days by set
                     membership per split, non-members written as nan for
                     'unbounded coords=discard'.
kfoldcv_folds.csv  : fold index and its validation error.
kfoldcv_k.csv      : k, bias of the k-fold CV estimate, its spread.
kfoldcv.png        : matplotlib preview (checking only).
"""

# /// script
# requires-python = ">=3.10"
# dependencies = ["matplotlib==3.10.7", "numpy==2.3.4"]
# ///

import numpy as np
import matplotlib

matplotlib.use("Agg")
import matplotlib.pyplot as plt

from pathlib import Path

OUT_DIR = Path(__file__).parent

report = []


def check(name, ok):
    report.append((name, bool(ok)))
    print(f"  [{'ok' if ok else 'FAIL'}] {name}")


DEGREE = 3
XLO, XHI = 0.0, 11.0                         # the plotted range of x


def cubic(xs, ys):
    """Minimum-norm least-squares polynomial of degree DEGREE. With three
    days and four coefficients the system is underdetermined, so a plain
    solve is not defined; lstsq picks the smallest-norm solution, which
    still passes exactly through the training days."""
    coef, *_ = np.linalg.lstsq(np.vander(xs, DEGREE + 1), ys, rcond=None)
    return coef


def sqerr(xs, ys, coef):
    return float(np.mean((np.polyval(coef, xs) - ys) ** 2))
    4 days: morning minimum 1, 6, 8, 10; maximum daytime 15, 18, 15, 23
    training error of the cubic on all four days: 2.07e-26

B-days

The four days, and the cubic learned from all four. With four points and four coefficients that fit is exact, so the training error is zero and says nothing about the risk — which is what makes a held-out day necessary.

x = np.array([1.0, 6.0, 8.0, 10.0])
y = np.array([15.0, 18.0, 15.0, 23.0])
m = len(x)
coef_full = cubic(x, y)
grid = np.linspace(XLO, XHI, 400)
print(f"    {m} days: morning minimum "
      + ", ".join(f"{v:.0f}" for v in x)
      + "; maximum daytime " + ", ".join(f"{v:.0f}" for v in y))
print(f"    training error of the cubic on all four days: "
      f"{sqerr(x, y, coef_full):.2e}")
check("[B-days]  the cubic on all four days has (almost) zero training error",
      sqerr(x, y, coef_full) < 1e-16)
  [ok] [B-days]  the cubic on all four days has (almost) zero training error
    holding out day 1 (x = 1): validation error 1.37, curve on [0, 11] runs 5.5..33.9

B-split

One split, the flattering one: hold out the first day. The cubic through the other three has zero training error and a validation error of 1.37, so it looks like a good hypothesis. Over the plotted range it runs from 5.5 to 33.9 degrees while the four labels lie between 15 and 23, and it departs by 13.0 degrees from a straight line through all four days: it is useless away from its three days, and the single split hid that.

# line_all is the reference a reader would accept as sane; the cubic of this
# split is judged by how far it wanders from it, not by its validation error
HELD = 0                                     # index of the held-out day
tr = np.array([i for i in range(m) if i != HELD])
coef_one = cubic(x[tr], y[tr])
err_one = (np.polyval(coef_one, x[HELD]) - y[HELD]) ** 2
curve_one = np.polyval(coef_one, grid)
print(f"    holding out day {HELD + 1} (x = {x[HELD]:.0f}): validation error "
      f"{err_one:.2f}, curve on [{XLO:.0f}, {XHI:.0f}] runs "
      f"{curve_one.min():.1f}..{curve_one.max():.1f}")
check("[B-split] the split's training error is zero: the cubic interpolates "
      "its three days",
      float(np.abs(np.polyval(coef_one, x[tr]) - y[tr]).max()) < 1e-8)
check("[B-split] its validation error is small, under 3",
      err_one < 3.0)
line_all = np.polyfit(x, y, 1)
gap = float(np.abs(curve_one - np.polyval(line_all, grid)).max())
print(f"    away from its three days it departs from a straight line that "
      f"fits all four by up to {gap:.1f} degrees")
check("[B-split] yet the hypothesis is useless away from its three days: it "
      "leaves the range of the observed labels on both sides",
      curve_one.min() < y.min() - 1.0 and curve_one.max() > y.max() + 1.0)
check("[B-split] and it departs from a line through all four days by more "
      "than ten degrees",
      gap > 10.0)
  [ok] [B-split] the split's training error is zero: the cubic interpolates its three days
  [ok] [B-split] its validation error is small, under 3
    away from its three days it departs from a straight line that fits all four by up to 13.0 degrees
  [ok] [B-split] yet the hypothesis is useless away from its three days: it leaves the range of the observed labels on both sides
  [ok] [B-split] and it departs from a line through all four days by more than ten degrees
    validation errors 1.37, 8.05, 7.02, 72.42; 4-fold CV estimate 22.22

B-cv

All four splits: validation errors 1.37, 8.05, 7.02 and 72.42, averaging to the 4-fold CV estimate 22.22. The flattering split understates it sixteenfold.

errs, coefs = [], []
for b in range(m):
    keep = np.array([i for i in range(m) if i != b])
    c = cubic(x[keep], y[keep])
    coefs.append(c)
    errs.append(float((np.polyval(c, x[b]) - y[b]) ** 2))
cv = float(np.mean(errs))
print("    validation errors " + ", ".join(f"{e:.2f}" for e in errs)
      + f"; 4-fold CV estimate {cv:.2f}")
check("[B-cv]    every split interpolates its own three days",
      all(float(np.abs(np.polyval(coefs[b],
                                  x[[i for i in range(m) if i != b]])
                       - y[[i for i in range(m) if i != b]]).max()) < 1e-8
          for b in range(m)))
check(f"[B-cv]    the flattering split understates the CV estimate "
      f"({cv:.2f}) more than tenfold",
      cv > 10 * err_one)
  [ok] [B-cv]    every split interpolates its own three days
  [ok] [B-cv]    the flattering split understates the CV estimate (22.22) more than tenfold

B-k

The choice of k, in a SEPARATE setting from the four days above: a linear model rather than a cubic, and 2000 datasets of fifteen days drawn INDEPENDENTLY from y = 4 + 0.8 x plus noise (not resampled from one dataset — independence is what lets this estimate a variance that cannot be read off a single dataset). For k = 2, 3, 5 and 15 (leave-one-out) the estimate's bias with respect to the expected loss falls with k, since each training set holds a fraction (k-1)/k of the days, and so does its spread across datasets.

rng = np.random.default_rng(0)
R, N, NOISE = 2000, 15, 1.5
ks = [2, 3, 5, 15]


def erm_line(xs, ys):
    return np.polyfit(xs, ys, 1)


def draw(n):
    xs = rng.uniform(-15.0, 5.0, n)
    return xs, 4.0 + 0.8 * xs + rng.normal(0.0, NOISE, n)


x_fresh, y_fresh = draw(20000)              # for the expected loss
est = {k: [] for k in ks}
true_risk = []
for _ in range(R):
    xs, ys = draw(N)
    true_risk.append(sqerr(x_fresh, y_fresh, erm_line(xs, ys)))
    for k in ks:
        f = np.arange(N) % k
        e = [sqerr(xs[f == b], ys[f == b], erm_line(xs[f != b], ys[f != b]))
             for b in range(k)]
        est[k].append(float(np.mean(e)))
risk = float(np.mean(true_risk))
rows = [(k, float(np.mean(est[k]) - risk), float(np.std(est[k]))) for k in ks]
bias = {r[0]: r[1] for r in rows}
check(f"[B-k]     expected loss of ERM on 15 days {risk:.2f}; bias of the "
      f"k-fold CV estimate "
      + ", ".join(f"k={r[0]}: {r[1]:+.2f}" for r in rows),
      bias[2] > bias[5] > bias[15] and bias[2] > 0.1 and abs(bias[15]) < 0.1)

# ---- CSV ------------------------------------------------------------------
with open(OUT_DIR / "kfoldcv_days.csv", "w") as fh:
    fh.write("x,y\n")
    for a, b_ in zip(x, y):
        fh.write(f"{a:.4f},{b_:.4f}\n")

with open(OUT_DIR / "kfoldcv_lines.csv", "w") as fh:
    fh.write("x,full," + ",".join(f"line{b + 1}" for b in range(m)) + "\n")
    for i, gx in enumerate(grid):
        vals = [np.polyval(coef_full, gx)] + [np.polyval(c, gx) for c in coefs]
        fh.write(f"{gx:.4f}," + ",".join(f"{v:.4f}" for v in vals) + "\n")

with open(OUT_DIR / "kfoldcv_panels.csv", "w") as fh:
    cols = []
    for b in range(m):
        cols += [f"train{b + 1}", f"val{b + 1}"]
    fh.write("x," + ",".join(cols) + "\n")
    for i in range(m):
        row = [f"{x[i]:.4f}"]
        for b in range(m):
            row.append("nan" if i == b else f"{y[i]:.4f}")   # train
            row.append(f"{y[i]:.4f}" if i == b else "nan")   # val
        fh.write(",".join(row) + "\n")

with open(OUT_DIR / "kfoldcv_folds.csv", "w") as fh:
    fh.write("fold,valerr\n")
    for b, e in enumerate(errs):
        fh.write(f"{b + 1},{e:.4f}\n")

with open(OUT_DIR / "kfoldcv_k.csv", "w") as fh:
    fh.write("k,bias,spread\n")
    for k, bi, sd in rows:
        fh.write(f"{k},{bi:.4f},{sd:.4f}\n")
  [ok] [B-k]     expected loss of ERM on 15 days 2.62; bias of the k-fold CV estimate k=2: +0.48, k=3: +0.20, k=5: +0.08, k=15: +0.01

8/8 checks pass
wrote kfoldcv_days.csv, kfoldcv_lines.csv, kfoldcv_panels.csv, kfoldcv_folds.csv, kfoldcv_k.csv, kfoldcv.png in /Users/junga1/dictionaryappliedml/pythondemos

B-preview

fig = plt.figure(figsize=(10.0, 5.6))
ax = fig.add_subplot(2, 1, 1)
ax.plot(grid, np.polyval(coef_full, grid), "-", color="black", lw=1.4,
        label="learned hypothesis")
ax.plot(x, y, "o", color="black", ms=7, label="data points")
ax.set_xlim(XLO, XHI)
ax.set_xlabel("morning minimum temperature $x$")
ax.set_ylabel("maximum daytime\ntemperature $y$")
ax.set_title("four days and the degree-3 hypothesis learned from all of them",
             fontsize=9)
ax.legend(frameon=False, fontsize=8)
for b in range(m):
    axb = fig.add_subplot(2, m, m + b + 1)
    keep = [i for i in range(m) if i != b]
    axb.plot(grid, np.polyval(coefs[b], grid), "-", color="black", lw=1.2)
    axb.plot(x[keep], y[keep], "o", color="black", ms=6)
    axb.plot(x[b], y[b], "s", color="red", mfc="none", ms=10, mew=2.0)
    axb.set_xlim(XLO, XHI)
    axb.set_ylim(min(0, min(np.polyval(c, grid).min() for c in coefs)) - 2,
                 max(np.polyval(c, grid).max() for c in coefs) + 2)
    axb.set_title(f"$b={b + 1}$, val. error {errs[b]:.2f}", fontsize=8)
    axb.set_xlabel("$x$", fontsize=8)
    if b == 0:
        axb.set_ylabel("$y$", fontsize=8)
    axb.tick_params(labelsize=7)
fig.tight_layout()
fig.savefig(OUT_DIR / "kfoldcv.png", dpi=110)

n_ok = sum(ok for _, ok in report)
print(f"\n{n_ok}/{len(report)} checks pass")
print(f"wrote kfoldcv_days.csv, kfoldcv_lines.csv, kfoldcv_panels.csv, "
      f"kfoldcv_folds.csv, kfoldcv_k.csv, kfoldcv.png in {OUT_DIR}")
if n_ok != len(report):
    raise SystemExit(1)
Preview figure produced by kfoldcv.py
The preview figure the block B-preview writes when the script runs