{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3",
   "language": "python"
  },
  "language_info": {
   "name": "python"
  },
  "colab": {
   "name": "testset.ipynb"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# test set \u2014 Python demo\n\nNumerical companion to the entry [test set](https://dictionaryofml.org/terms/testset.html) of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org/): it recomputes what the entry states and prints one line per check.\n\nThe Krems weather narrative of the entry: today's maximum temperature is the feature, the next day's maximum temperature the label, and the data generation is a known linear rule plus Gaussian noise, so the risk of any hypothesis is computable and every claim of the entry can be checked against it. One block per paragraph of the entry (marked [P...]), in order. Self-contained (numpy/matplotlib only), fixed seeds.\n\nRequires NumPy and Matplotlib only, and uses fixed seeds, so the printed numbers reproduce exactly. Generated from [`pythondemos/testset.py`](https://dictionaryofml.org/terms/testset.py); CC BY 4.0."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "# Notebook shim: the script resolves output paths relative to __file__,\n# which a notebook kernel does not define; everything lands in the\n# working directory instead.\nimport os\n__file__ = os.path.join(os.getcwd(), \"testset.py\")\nos.makedirs(\"pythondemos\", exist_ok=True)"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "\"\"\"\ntestset.py -- numerical companion to the entry 'test set'.\n\nThe Krems weather narrative of the entry: today's maximum temperature is\nthe feature, the next day's maximum temperature the label, and the data\ngeneration is a known linear rule plus Gaussian noise, so the risk of\nany hypothesis is computable and every claim of the entry can be checked\nagainst it. One block per paragraph of the entry (marked [P...]), in\norder. Self-contained (numpy/matplotlib only), fixed seeds.\n\nBlocks\n------\n[P-tuning]  Training and validation data points tune the learned\n            hypothesis: the average loss on the points used for\n            training and model selection understates the loss on fresh\n            data points.\n[P-def]     The definition's estimator claims: for a hypothesis learned\n            and selected without reference to the test set, the test\n            error is an unbiased estimator of the risk (mean over many\n            fresh test sets matches the analytic risk); the validation\n            error of the selected hypothesis is optimistic (its mean\n            lies below the winner's risk), while the test error is not.\n[P-size]    The concentration bound on the test-set size: for a loss\n            with values in [0,1], the deviation of the test error from\n            the risk stays within sqrt(log(2/delta)/(2 m)) in at least\n            a 1-delta fraction of repeated test draws, and the typical\n            deviation shrinks like 1/sqrt(m); the entry's typical\n            50/25/25 split of 200 data points is printed.\n[P-misuse]  Evaluating many candidate hypotheses on the test set and\n            keeping the best turns the test set into a validation set:\n            the reported minimum test error lies below the winner's\n            risk, while a fresh test set restores an honest estimate.\n\nOutputs\n-------\ntestset.png : preview figure (checking only).\n\"\"\"\n\nimport numpy as np\nimport matplotlib\n\nmatplotlib.use(\"Agg\")\nimport matplotlib.pyplot as plt\n\nfrom math import erf\nfrom pathlib import Path\n\nOUT_DIR = Path(__file__).parent\n\nreport = []\n\n\ndef check(name, ok):\n    report.append((name, bool(ok)))\n    print(f\"  [{'ok' if ok else 'FAIL'}] {name}\")\n\n\n# data generation: next day's maximum temperature from today's maximum\nW0, W1, SIGMA = 1.5, 0.9, 2.5\nX_LO, X_HI = -5.0, 25.0\nXGRID = np.linspace(X_LO, X_HI, 20001)\n\n\ndef days(n, gen):\n    x = gen.uniform(X_LO, X_HI, n)\n    return x, W0 + W1 * x + gen.normal(0.0, SIGMA, n)\n\n\ndef sqerr(x, y, coeffs):\n    return float(np.mean((y - np.polyval(coeffs, x)) ** 2))\n\n\ndef risk_sq(coeffs):\n    \"\"\"Risk under squared error: mean squared bias plus noise variance.\"\"\"\n    d = (W0 + W1 * XGRID) - np.polyval(coeffs, XGRID)\n    return float(np.mean(d ** 2)) + SIGMA ** 2\n\n\nPHI = np.vectorize(lambda t: 0.5 * (1.0 + erf(t / np.sqrt(2.0))))\nBAND = 3.0\n\n\ndef banderr(x, y, coeffs):\n    \"\"\"0/1 loss: 1 if the prediction misses the label by more than BAND.\"\"\"\n    return float(np.mean(np.abs(y - np.polyval(coeffs, x)) > BAND))\n\n\ndef risk_band(coeffs):\n    d = (W0 + W1 * XGRID) - np.polyval(coeffs, XGRID)\n    inside = PHI((BAND - d) / SIGMA) - PHI((-BAND - d) / SIGMA)\n    return float(np.mean(1.0 - inside))"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[P-tuning]** Training and validation data points tune the learned hypothesis: the average loss on the points used for training and model selection understates the loss on fresh data points."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[P-tuning] the data points used for training and selection tune \"\n      \"the learned hypothesis\")\n\ngen = np.random.default_rng(7)\nx_tr, y_tr = days(10, gen)\nx_va, y_va = days(5, gen)\nc_lin = np.polyfit(x_tr, y_tr, 1)\nc_pol = np.polyfit(x_tr, y_tr, 3)\nc_sel = c_lin if sqerr(x_va, y_va, c_lin) <= sqerr(x_va, y_va, c_pol) \\\n    else c_pol\nused = sqerr(np.r_[x_tr, x_va], np.r_[y_tr, y_va], c_sel)\nprint(f\"    average loss on the used data points {used:.2f}, \"\n      f\"risk {risk_sq(c_sel):.2f}\")\ncheck(\"the used data points make the hypothesis look better than fresh ones\",\n      used < risk_sq(c_sel))"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[P-def]** The definition's estimator claims: for a hypothesis learned and selected without reference to the test set, the test error is an unbiased estimator of the risk (mean over many fresh test sets matches the analytic risk); the validation error of the selected hypothesis is optimistic (its mean lies below the winner's risk), while the test error is not."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"\\n[P-def] the test error is an unbiased estimator of the risk; \"\n      \"the validation error is not\")\n\ngen = np.random.default_rng(17)\ntesterrs = [sqerr(*days(25, gen), c_sel) for _ in range(3000)]\nmean_test, r_sel = float(np.mean(testerrs)), risk_sq(c_sel)\nse = float(np.std(testerrs)) / np.sqrt(3000.0)\nprint(f\"    mean test error over 3000 fresh test sets {mean_test:.3f}, \"\n      f\"risk {r_sel:.3f}\")\ncheck(\"mean test error matches the risk (within 3 standard errors)\",\n      abs(mean_test - r_sel) < 3 * se)\n\ngen = np.random.default_rng(27)\ngap_val, gap_test = [], []\nfor _ in range(400):\n    xr, yr = days(20, gen)\n    xv, yv = days(10, gen)\n    cands = [np.polyfit(xr, yr, deg) for deg in range(5)]\n    verrs = [sqerr(xv, yv, c) for c in cands]\n    win = cands[int(np.argmin(verrs))]\n    gap_val.append(min(verrs) - risk_sq(win))\n    gap_test.append(sqerr(*days(25, gen), win) - risk_sq(win))\nprint(f\"    winner's validation error minus risk, averaged: \"\n      f\"{np.mean(gap_val):.2f}; test error minus risk: \"\n      f\"{np.mean(gap_test):.2f}\")\ncheck(\"the validation error of the selected hypothesis is optimistic\",\n      np.mean(gap_val) < -0.5)\ncheck(\"the test error of the selected hypothesis is not\",\n      abs(np.mean(gap_test)) < 0.2)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[P-size]** The concentration bound on the test-set size: for a loss with values in [0,1], the deviation of the test error from the risk stays within sqrt(log(2/delta)/(2 m)) in at least a 1-delta fraction of repeated test draws, and the typical deviation shrinks like 1/sqrt(m); the entry's typical 50/25/25 split of 200 data points is printed."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"\\n[P-size] the concentration bound sizes the test set\")\n\nDELTA = 0.05\nr_band = risk_band(c_sel)\nrms = {}\ngen = np.random.default_rng(37)\nfor m in (25, 100, 400):\n    devs = np.array([banderr(*days(m, gen), c_sel) - r_band\n                     for _ in range(3000)])\n    rms[m] = float(np.sqrt(np.mean(devs ** 2)))\n    if m == 100:\n        bnd = np.sqrt(np.log(2.0 / DELTA) / (2.0 * m))\n        cover = float(np.mean(np.abs(devs) <= bnd))\n        print(f\"    m = {m}: bound {bnd:.3f}, deviation within the bound \"\n              f\"in {100 * cover:.1f}% of draws\")\n        check(\"the deviation stays within the bound in at least 95% of draws\",\n              cover >= 1.0 - DELTA)\nprint(f\"    rms deviation: m=100 gives {rms[100]:.4f}, \"\n      f\"m=400 gives {rms[400]:.4f}\")\ncheck(\"the typical deviation shrinks like one over sqrt of the size\",\n      abs(rms[100] / rms[400] - 2.0) < 0.5)\nm_all = 200\nprint(f\"    typical split of {m_all} data points: \"\n      f\"{m_all // 2} training, {m_all // 4} validation, {m_all // 4} test\")"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[P-misuse]** Evaluating many candidate hypotheses on the test set and keeping the best turns the test set into a validation set: the reported minimum test error lies below the winner's risk, while a fresh test set restores an honest estimate."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"\\n[P-misuse] keeping the best of many candidates turns the test \"\n      \"set into a validation set\")\n\ngen = np.random.default_rng(47)\ngap_reported, gap_fresh = [], []\nfor _ in range(300):\n    cands = [np.array([W1 + gen.normal(0.0, 0.15),\n                       W0 + gen.normal(0.0, 1.5)]) for _ in range(40)]\n    xt, yt = days(20, gen)\n    terrs = [sqerr(xt, yt, c) for c in cands]\n    win = cands[int(np.argmin(terrs))]\n    gap_reported.append(min(terrs) - risk_sq(win))\n    gap_fresh.append(sqerr(*days(20, gen), win) - risk_sq(win))\nprint(f\"    reported minimum test error minus the winner's risk, averaged: \"\n      f\"{np.mean(gap_reported):.2f}; on a fresh test set: \"\n      f\"{np.mean(gap_fresh):.2f}\")\ncheck(\"the reported minimum test error understates the winner's risk\",\n      np.mean(gap_reported) < -0.5)\ncheck(\"a fresh test set restores an honest estimate\",\n      abs(np.mean(gap_fresh)) < 0.3)\n\n# ------------------------------------------------------------------ preview\nfig, axes = plt.subplots(1, 2, figsize=(9.6, 3.6))\nax = axes[0]\nxs = np.linspace(X_LO, X_HI, 200)\nax.plot(xs, np.polyval(c_lin, xs), \"-\", color=\"black\", lw=1.2,\n        label=\"linear hypothesis (selected)\")\nax.plot(xs, np.polyval(c_pol, xs), \"--\", color=\"black\", lw=1.2,\n        label=\"degree-3 hypothesis\")\nax.plot(x_tr, y_tr, \"o\", ms=6, color=\"black\", label=\"training set\")\nax.plot(x_va, y_va, \"^\", ms=7, markerfacecolor=\"none\",\n        markeredgecolor=\"black\", label=\"validation set\")\nx_te, y_te = days(5, np.random.default_rng(57))\nax.plot(x_te, y_te, \"s\", ms=6, markerfacecolor=\"none\",\n        markeredgecolor=\"black\", label=\"test set\")\nax.set_xlabel(\"today's maximum temperature\")\nax.set_ylabel(\"next day's maximum temperature\")\nax.set_title(\"the test set enters neither training nor selection\",\n             fontsize=9)\nax.legend(frameon=False, fontsize=7)\n\nax = axes[1]\nms = np.array(sorted(rms))\nax.loglog(ms, [rms[m] for m in ms], \"o-\", color=\"black\", lw=1.2,\n          label=\"rms deviation of the test error\")\nax.loglog(ms, np.sqrt(np.log(2.0 / DELTA) / (2.0 * ms)), \"--\",\n          color=\"black\", lw=1.2, label=\"concentration bound (delta = 0.05)\")\nax.set_xlabel(\"test-set size\")\nax.set_ylabel(\"deviation from the risk\")\nax.set_title(\"the deviation shrinks like one over sqrt of the size\",\n             fontsize=9)\nax.legend(frameon=False, fontsize=8)\n\nfig.tight_layout()\nfig.savefig(OUT_DIR / \"testset.png\", dpi=110)\n\nn_ok = sum(ok for _, ok in report)\nprint(f\"\\n{n_ok}/{len(report)} checks pass\")\nprint(\"wrote testset.png\")\nif n_ok != len(report):\n    raise SystemExit(1)"
  }
 ]
}