{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3",
   "language": "python"
  },
  "language_info": {
   "name": "python"
  },
  "colab": {
   "name": "hyperparameter.ipynb"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# hyperparameter \u2014 Python demo\n\nNumerical companion to the entry [hyperparameter](https://dictionaryofml.org/terms/hyperparameter.html) of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org/): it recomputes what the entry states and prints one line per check.\n\nShows why a hyperparameter is chosen by the validation error and not by the training error: for a polynomial model, the training error falls with every increase of the degree, while the validation error falls and then rises again, so only the validation error selects a degree. Self-contained (numpy/matplotlib only), fixed seed.\n\nRequires NumPy and Matplotlib only, and uses fixed seeds, so the printed numbers reproduce exactly. Generated from [`pythondemos/hyperparameter.py`](https://dictionaryofml.org/terms/hyperparameter.py); CC BY 4.0."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "# Notebook shim: the script resolves output paths relative to __file__,\n# which a notebook kernel does not define; everything lands in the\n# working directory instead.\nimport os\n__file__ = os.path.join(os.getcwd(), \"hyperparameter.py\")\nos.makedirs(\"pythondemos\", exist_ok=True)"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "\"\"\"\nhyperparameter.py \u2014 numerical companion to the glossary entry\n'hyperparameter'.\n\nPurpose\n-------\nShows why a hyperparameter is chosen by the validation error and not by\nthe training error: for a polynomial model, the training error falls\nwith every increase of the degree, while the validation error falls and\nthen rises again, so only the validation error selects a degree.\nSelf-contained (numpy/matplotlib only), fixed seed.\n\nSetup\n-----\nTraining set: m = 20 data points with feature x drawn uniformly from\n[0, 1] and label y = sin(2 pi x) + Gaussian noise scaled by 0.3;\nvalidation set: 100 further points from the same source.  Model: a\npolynomial of degree p in x, with the degree as the hyperparameter;\nthe model parameters are the coefficients, learned by least squares\n(ERM with the squared error) on the training set.  Degrees 0 to\n9 are compared by their average squared error on the training set\n(training error) and on the validation set (validation error).\n\nBlocks\n------\n[B-data]   The training set and the validation set.\n[B-sweep]  Training and validation error for each degree: the training\n           error never increases with the degree, the validation error\n           has an interior minimum, and the degree that minimizes it\n           (3) is not the degree that minimizes the training error (9).\n\nOutputs\n-------\nhyperparameter_errors.csv : degree, trainerr, valerr.\nhyperparameter.png        : matplotlib preview of the entry's figure\n                            (checking only).\n\"\"\"\n\nimport numpy as np\nimport matplotlib\n\nmatplotlib.use(\"Agg\")\nimport matplotlib.pyplot as plt\n\nfrom pathlib import Path\n\nOUT_DIR = Path(__file__).parent\n\nreport = []\n\n\ndef check(name, ok):\n    report.append((name, bool(ok)))\n    print(f\"  [{'ok' if ok else 'FAIL'}] {name}\")\n\n\nrng = np.random.default_rng(1)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-data]** The training set and the validation set."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "m, m_val, sigma = 20, 100, 0.3\nx_tr = np.sort(rng.uniform(0.0, 1.0, m))\ny_tr = np.sin(2 * np.pi * x_tr) + sigma * rng.standard_normal(m)\nx_va = np.sort(rng.uniform(0.0, 1.0, m_val))\ny_va = np.sin(2 * np.pi * x_va) + sigma * rng.standard_normal(m_val)\ncheck(f\"[B-data]  training set of {m} and validation set of {m_val} points\",\n      len(x_tr) == m and len(x_va) == m_val)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-sweep]** Training and validation error for each degree: the training error never increases with the degree, the validation error has an interior minimum, and the degree that minimizes it (3) is not the degree that minimizes the training error (9)."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "degrees = np.arange(0, 10)\ntr_err, va_err = [], []\nfor p in degrees:\n    A = np.vander(x_tr, p + 1)                     # least squares on the trainset\n    w = np.linalg.lstsq(A, y_tr, rcond=None)[0]\n    tr_err.append(float(np.mean((y_tr - A @ w) ** 2)))\n    va_err.append(float(np.mean((y_va - np.vander(x_va, p + 1) @ w) ** 2)))\ntr_err, va_err = np.array(tr_err), np.array(va_err)\np_tr, p_va = int(degrees[np.argmin(tr_err)]), int(degrees[np.argmin(va_err)])\nprint(\"  degree -> training error / validation error: \" +\n      \", \".join(f\"{p}: {a:.3f}/{b:.3f}\" for p, a, b in zip(degrees, tr_err, va_err)))\ncheck(\"[B-sweep] the training error never increases with the degree\",\n      np.all(np.diff(tr_err) <= 1e-12))\ncheck(f\"[B-sweep] the validation error has an interior minimum at degree \"\n      f\"{p_va}\", 0 < p_va < degrees[-1])\ncheck(f\"[B-sweep] the training error would pick degree {p_tr}, the \"\n      f\"validation error degree {p_va}\", p_tr == 9 and p_va == 3)\n\n# ---------------------------------------------------------------- CSV\nwith open(OUT_DIR / \"hyperparameter_errors.csv\", \"w\") as fh:\n    fh.write(\"degree,trainerr,valerr\\n\")\n    for p, a, b in zip(degrees, tr_err, va_err):\n        fh.write(f\"{p},{a:.4f},{b:.4f}\\n\")\n\n# -------------------------------------------------------------- preview\nfig, ax = plt.subplots(figsize=(5.0, 3.6))\nax.plot(degrees, tr_err, \"k-\", marker=\"o\", ms=4, label=\"training error\")\nax.plot(degrees, va_err, \"k--\", marker=\"s\", ms=4, mfc=\"none\",\n        label=\"validation error\")\nax.axvline(p_va, color=\"0.6\", ls=\":\", lw=1)\nax.set_yscale(\"log\")\nax.set_xlabel(\"polynomial degree (the hyperparameter)\")\nax.set_ylabel(\"average squared error\")\nax.set_title(\"the validation error selects the degree, the training error cannot\")\nax.legend(frameon=False, fontsize=8)\nfig.tight_layout()\nfig.savefig(OUT_DIR / \"hyperparameter.png\", dpi=110)\n\nn_ok = sum(ok for _, ok in report)\nprint(f\"\\n{n_ok}/{len(report)} checks pass\")\nprint(f\"wrote {OUT_DIR / 'hyperparameter_errors.csv'}, \"\n      f\"{OUT_DIR / 'hyperparameter.png'}\")\nif n_ok != len(report):\n    raise SystemExit(1)"
  }
 ]
}