{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3",
   "language": "python"
  },
  "language_info": {
   "name": "python"
  },
  "colab": {
   "name": "valset.ipynb"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# validation set \u2014 Python demo\n\nNumerical companion to the entry [validation set](https://dictionaryofml.org/terms/valset.html) of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org/): it recomputes what the entry states and prints one line per check.\n\nThe weather narrative of the 'training set' entry, continued: the same three training days and the same two hypotheses (straight line, degree-two polynomial), plus three validation days held back from model training. The validation error ranks the two hypotheses opposite to the training error, the hypothesis with the smaller validation error is selected (model selection), and three further test days assess the selected hypothesis. The demo also evaluates the Hoeffding lower bound on the validation-set size behind the entry's third figure. Self-contained (numpy/matplotlib only), deterministic.\n\nRequires NumPy and Matplotlib only, and uses fixed seeds, so the printed numbers reproduce exactly. Generated from [`pythondemos/valset.py`](https://dictionaryofml.org/terms/valset.py); CC BY 4.0."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "# Notebook shim: the script resolves output paths relative to __file__,\n# which a notebook kernel does not define; everything lands in the\n# working directory instead.\nimport os\n__file__ = os.path.join(os.getcwd(), \"valset.py\")\nos.makedirs(\"pythondemos\", exist_ok=True)"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "\"\"\"\nvalset.py -- numerical companion to the entry 'validation set'.\n\nThe weather narrative of the 'training set' entry, continued: the same\nthree training days and the same two hypotheses (straight line,\ndegree-two polynomial), plus three validation days held back from model\ntraining.  The validation error ranks the two hypotheses opposite to\nthe training error, the hypothesis with the smaller validation error is\nselected (model selection), and three further test days assess the\nselected hypothesis.  The demo also evaluates the Hoeffding lower bound\non the validation-set size behind the entry's third figure.\nSelf-contained (numpy/matplotlib only), deterministic.\n\nBlocks\n------\n[B-data]     Recreate the three training days of the 'training set'\n             entry and learn the two hypotheses from them; check that\n             the fitted coefficients match those of trainset.py, so the\n             figures of the two entries show the same curves.\n[B-valset]   Three validation days held back from model training. The\n             validation error of the polynomial far exceeds that of the\n             line, reversing the training-error ranking.\n[B-modelsel] Select the hypothesis with the smaller validation error\n             (the line); evaluate the selected hypothesis on three test\n             days that entered neither training nor selection.\n[B-bound]    The Hoeffding lower bound ln(2/delta) / (2 Delta^2) on the\n             validation-set size: check the entry's example (185 data\n             points for Delta = 0.1, delta = 0.05) and print the bound\n             for the three curves of the entry's figure.\n[B-kfold]    3-fold cross-validation on the six pooled days: each fold\n             yields a noisy validation error and their average is the CV\n             estimate. Over 300 replicate datasets, the CV estimate\n             varies far less than a single split with folds of the same\n             size, backing the entry's remedy for scarce data points.\n\nOutputs\n-------\npythondemos/valset.png          : preview figure (checking only).\npythondemos/valset_train.csv    : the three training days (x = morning\n                                  minimum temperature, y = maximum\n                                  daytime temperature)\npythondemos/valset_val.csv      : the three validation days\npythondemos/valset_test.csv     : the three test days\npythondemos/valset_line.csv     : the fitted line on a two-point grid\npythondemos/valset_poly.csv     : the fitted polynomial on a dense grid\n\"\"\"\n\nimport numpy as np\nimport matplotlib\n\nmatplotlib.use(\"Agg\")\nimport matplotlib.pyplot as plt\n\nfrom pathlib import Path\n\nOUT_DIR = Path(__file__).parent\n\nreport = []\n\n\ndef check(name, ok):\n    report.append((name, bool(ok)))\n    print(f\"  [{'ok' if ok else 'FAIL'}] {name}\")\n\n\ndef days(n, gen, lo=-15.0, hi=5.0, w1=0.8, w0=4.0, noise=2.0):\n    \"\"\"n days: morning minimum x and maximum daytime temperature y.\"\"\"\n    x = np.sort(gen.uniform(lo, hi, n))\n    y = w0 + w1 * x + gen.normal(0.0, noise, n)\n    return x, y\n\n\ndef avg_sqerr(x, y, coeffs):\n    return float(np.mean((y - np.polyval(coeffs, x)) ** 2))"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-data]** Recreate the three training days of the 'training set' entry and learn the two hypotheses from them; check that the fitted coefficients match those of trainset.py, so the figures of the two entries show the same curves."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-data] the three training days and the two hypotheses of the \"\n      \"'training set' entry\")\n\nx, y = days(3, np.random.default_rng(100))\nc_line = np.polyfit(x, y, 1)\nc_poly = np.polyfit(x, y, 2)\ntrainerr_line = avg_sqerr(x, y, c_line)\ntrainerr_poly = avg_sqerr(x, y, c_poly)\nprint(f\"    fitted line: max temperature = {c_line[1]:.2f} + {c_line[0]:.2f}\"\n      f\" * min temperature; training error {trainerr_line:.3f}\")\ncheck(\"the fitted line matches trainset.py (same coefficients)\",\n      abs(c_line[0] - 0.8688) < 5e-3 and abs(c_line[1] - 4.6785) < 5e-3)\ncheck(\"the polynomial's training error is zero (up to round-off)\",\n      trainerr_poly < 1e-10)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-valset]** Three validation days held back from model training. The validation error of the polynomial far exceeds that of the line, reversing the training-error ranking."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"\\n[B-valset] three validation days held back from model training\")\n\nxv, yv = days(3, np.random.default_rng(1100))\nfor xr, yr in zip(xv, yv):\n    print(f\"    morning minimum {xr:6.1f} C, maximum daytime {yr:6.1f} C\")\nvalerr_line = avg_sqerr(xv, yv, c_line)\nvalerr_poly = avg_sqerr(xv, yv, c_poly)\nprint(f\"    validation error: line {valerr_line:.2f}, \"\n      f\"polynomial {valerr_poly:.2f}\")\ncheck(\"the validation days are disjoint from the training days\",\n      len(set(np.round(xv, 6)) & set(np.round(x, 6))) == 0)\ncheck(\"the validation error reverses the training-error ranking\",\n      trainerr_poly < trainerr_line and valerr_poly > valerr_line)\ncheck(\"the polynomial's validation error far exceeds the line's\",\n      valerr_poly > 3 * valerr_line)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-modelsel]** Select the hypothesis with the smaller validation error (the line); evaluate the selected hypothesis on three test days that entered neither training nor selection."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"\\n[B-modelsel] the validation error selects the line; test days \"\n      \"assess it\")\n\nselected = \"line\" if valerr_line < valerr_poly else \"polynomial\"\nxt, yt = days(3, np.random.default_rng(3100))\ntesterr_line = avg_sqerr(xt, yt, c_line)\nprint(f\"    selected hypothesis: {selected}; \"\n      f\"test error of the line {testerr_line:.2f}\")\ncheck(\"the smaller validation error selects the line\", selected == \"line\")\ncheck(\"the test days entered neither training nor selection\",\n      len(set(np.round(xt, 6)) & set(np.round(np.r_[x, xv], 6))) == 0)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-bound]** The Hoeffding lower bound ln(2/delta) / (2 Delta^2) on the validation-set size: check the entry's example (185 data points for Delta = 0.1, delta = 0.05) and print the bound for the three curves of the entry's figure."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"\\n[B-bound] the Hoeffding lower bound on the validation-set size\")\n\n\ndef bound(delta, band):\n    return np.log(2.0 / delta) / (2.0 * band ** 2)\n\n\nprint(f\"    Delta = 0.1, delta = 0.05: {bound(0.05, 0.1):.1f} \"\n      f\"-> 185 data points\")\nfor delta in (0.01, 0.1, 0.5):\n    print(f\"    delta = {delta}: bound at Delta = 0.1 is \"\n          f\"{bound(delta, 0.1):.0f}\")\ncheck(\"the entry's example holds: 185 data points suffice\",\n      int(np.ceil(bound(0.05, 0.1))) == 185)\ncheck(\"the bound grows as the band narrows (0.05 needs 4x more than 0.1)\",\n      abs(bound(0.05, 0.05) / bound(0.05, 0.1) - 4.0) < 1e-12)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-kfold]** 3-fold cross-validation on the six pooled days: each fold yields a noisy validation error and their average is the CV estimate. Over 300 replicate datasets, the CV estimate varies far less than a single split with folds of the same size, backing the entry's remedy for scarce data points."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"\\n[B-kfold] averaging noisy per-fold validation errors is more \"\n      \"reliable\")\n\nX_all, Y_all = np.r_[x, xv], np.r_[y, yv]\n\n\ndef cv3(xs, ys, gen):\n    \"\"\"Per-fold validation errors of 3-fold CV of the linear model.\"\"\"\n    folds = gen.permutation(len(xs)).reshape(3, 2)\n    errs = []\n    for hold in folds:\n        keep = np.ones(len(xs), bool)\n        keep[hold] = False\n        c = np.polyfit(xs[keep], ys[keep], 1)\n        errs.append(avg_sqerr(xs[hold], ys[hold], c))\n    return errs\n\n\nfold_errs = cv3(X_all, Y_all, np.random.default_rng(5))\nprint(f\"    per-fold validation errors: \"\n      f\"{', '.join(f'{v:.2f}' for v in fold_errs)}; \"\n      f\"average {np.mean(fold_errs):.2f}\")\ncheck(\"the per-fold validation errors are noisy (spread over a 3x range)\",\n      max(fold_errs) > 3 * min(fold_errs))\n\ngen = np.random.default_rng(9)\nsingles, cv_avgs = [], []\nfor _ in range(300):\n    xr, yr = days(6, gen)\n    c = np.polyfit(xr[:4], yr[:4], 1)\n    singles.append(avg_sqerr(xr[4:], yr[4:], c))\n    cv_avgs.append(float(np.mean(cv3(xr, yr, gen))))\nstd_single, std_cv = float(np.std(singles)), float(np.std(cv_avgs))\nprint(f\"    over 300 replicate datasets: std of a single split \"\n      f\"{std_single:.1f}, std of the 3-fold CV average {std_cv:.1f}\")\ncheck(\"the CV average varies far less than a single split\",\n      std_single > 2 * std_cv)\n\n# --------------------------------------------------- outputs: CSVs, preview\nxs_line = np.array([-14.0, 5.0])\nxs_poly = np.linspace(-14.0, 5.0, 200)\n\n\ndef write_csv(name, cols, header):\n    np.savetxt(OUT_DIR / name, np.column_stack(cols), delimiter=\",\",\n               header=header, comments=\"\", fmt=\"%.4f\")\n\n\nwrite_csv(\"valset_train.csv\", (x, y), \"x,y\")\nwrite_csv(\"valset_val.csv\", (xv, yv), \"x,y\")\nwrite_csv(\"valset_test.csv\", (xt, yt), \"x,y\")\nwrite_csv(\"valset_line.csv\", (xs_line, np.polyval(c_line, xs_line)),\n          \"x,yhat\")\nwrite_csv(\"valset_poly.csv\", (xs_poly, np.polyval(c_poly, xs_poly)),\n          \"x,yhat\")\n\nfig, axes = plt.subplots(1, 2, figsize=(9.6, 3.6))\nax = axes[0]\nax.plot(xs_poly, np.polyval(c_poly, xs_poly), \"--\", color=\"black\", lw=1.2,\n        label=\"polynomial\")\nax.plot(xs_line, np.polyval(c_line, xs_line), \"-\", color=\"black\", lw=1.2,\n        label=\"line (selected)\")\nax.plot(x, y, \"o\", ms=6, color=\"black\", label=\"training set\")\nax.plot(xv, yv, \"^\", ms=7, markerfacecolor=\"none\", markeredgecolor=\"black\",\n        label=\"validation set\")\nax.plot(xt, yt, \"s\", ms=6, markerfacecolor=\"none\", markeredgecolor=\"black\",\n        label=\"test set\")\nax.set_xlabel(\"morning minimum temperature\")\nax.set_ylabel(\"maximum daytime temperature\")\nax.set_title(\"validation days rank the two hypotheses\", fontsize=9)\nax.legend(frameon=False, fontsize=7)\n\nax = axes[1]\nband = np.linspace(0.02, 0.5, 200)\nfor delta, style in ((0.01, \"-\"), (0.1, \"--\"), (0.5, \":\")):\n    ax.semilogy(band, bound(delta, band), style, color=\"black\", lw=1.2,\n                label=f\"confidence level {1 - delta:g}\")\nax.set_xlabel(\"uncertainty band half-width\")\nax.set_ylabel(\"required validation-set size\")\nax.set_title(\"Hoeffding lower bound on the validation-set size\", fontsize=9)\nax.legend(frameon=False, fontsize=8)\n\nfig.tight_layout()\nfig.savefig(OUT_DIR / \"valset.png\", dpi=110)\n\nn_ok = sum(ok for _, ok in report)\nprint(f\"\\n{n_ok}/{len(report)} checks pass\")\nprint(\"wrote valset.png\")\nif n_ok != len(report):\n    raise SystemExit(1)"
  }
 ]
}