{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3",
   "language": "python"
  },
  "language_info": {
   "name": "python"
  },
  "colab": {
   "name": "supervisedlearning.ipynb"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# supervised learning \u2014 Python demo\n\nNumerical companion to the entry [supervised learning](https://dictionaryofml.org/terms/supervisedlearning.html) of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org/): it recomputes what the entry states and prints one line per check.\n\nNumerical companion to the glossary entry 'supervisedlearning'. The GeoSphere Austria weather station Krems (station id 3805) records the minimum and the maximum air temperature of each day; this script downloads the records for 2024 from the GeoSphere data hub (dataset klima-v2-1d) and writes them to supervisedlearning_weather.csv. Each day is a data point. Its feature is the morning minimum temperature, its label the maximum temperature of that day. Both numbers are measured, so every data point of the training set carries its label, which is what makes the setting supervised.\n\nRequires NumPy and Matplotlib only, and uses fixed seeds, so the printed numbers reproduce exactly. Generated from [`pythondemos/supervisedlearning.py`](https://dictionaryofml.org/terms/supervisedlearning.py); CC BY 4.0."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "# Notebook shim: the script resolves output paths relative to __file__,\n# which a notebook kernel does not define; everything lands in the\n# working directory instead.\nimport os\n__file__ = os.path.join(os.getcwd(), \"supervisedlearning.py\")\nos.makedirs(\"pythondemos\", exist_ok=True)"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "\"\"\"Supervised learning on a year of days at Krems: each day carries a\nlabel, ERM fits a hypothesis to the labeled days, and the same labels\nserve a regression and a classification problem.\n\nPurpose\n-------\nNumerical companion to the glossary entry 'supervisedlearning'.  The\nGeoSphere Austria weather station Krems (station id 3805) records the\nminimum and the maximum air temperature of each day; this script\ndownloads the records for 2024 from the GeoSphere data hub (dataset\nklima-v2-1d) and writes them to supervisedlearning_weather.csv.  Each\nday is a data point.  Its feature is the morning minimum temperature,\nits label the maximum temperature of that day.  Both numbers are\nmeasured, so every data point of the training set carries its label,\nwhich is what makes the setting supervised.\n\nThe demo checks the entry's claims: (1) ERM over the linear model fits\na hypothesis whose training error is far below the sample variance of\nthe labels, and whose validation error on held-out days is close to it,\nso the labels support a prediction for days outside the training set;\n(2) labels alone do not guarantee this -- a degree-12 polynomial fitted\nto ten days attains a smaller training error and a much larger\nvalidation error; (3) the label space decides the learning task: with\nthe numeric maximum temperature as label the problem is regression,\nwith the binary \"frost in the morning\" label constructed from the same\nrecords it is classification, and the same ERM machinery applies with a\ndifferent loss.\n\nDeterministic: the data are a fixed archive year and every fit is computed in closed form.  Self-contained: numpy +\nmatplotlib only (stdlib urllib for the download).\n\nBlocks\n------\n[B-data]     Download the 366 daily temperature pairs at Krems for 2024;\n             feature = morning minimum, label = maximum of the day;\n             split into a training set (January to August) and a\n             validation set (September to December).\n[B-erm]      ERM over the linear model on the training set; check the\n             training error is far below the sample\n             variance of the labels, that the validation error is close\n             to the training error, and that both beat predicting the\n             sample mean of the training labels.\n[B-overfit]  The same labels, a training set of ten days and a\n             degree-12 polynomial: smaller training error, far larger\n             validation error.\n[B-classify] The label space decides the task: the binary label \"frost\n             in the morning\" turns the same records into a\n             classification problem; a threshold rule fitted by\n             ERM with the zero-one loss beats predicting the majority class.\n[B-plot]     Preview: the labeled days with the learned hypothesis, and\n             the overfitting polynomial beside it.\n\nOutputs\n-------\nsupervisedlearning_weather.csv : date, tmin, tmax for the 366 days\nsupervisedlearning_train.csv   : x, y of the training days\nsupervisedlearning_val.csv     : x, y of the validation days\nsupervisedlearning_fit.csv     : x, linear, poly -- the two hypotheses\nsupervisedlearning.png         : preview (checking only)\n\"\"\"\n\nimport json\nimport urllib.request\nfrom pathlib import Path\n\nimport numpy as np\nimport matplotlib\n\nmatplotlib.use(\"Agg\")\nimport matplotlib.pyplot as plt\n\nOUT_DIR = Path(__file__).parent\n\nreport = []\n\n\ndef check(name, ok):\n    report.append((name, bool(ok)))\n    print(f\"  [{'ok' if ok else 'FAIL'}] {name}\")"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-data]** Download the 366 daily temperature pairs at Krems for 2024; feature = morning minimum, label = maximum of the day; split into a training set (January to August) and a validation set (September to December)."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "URL = (\"https://dataset.api.hub.geosphere.at/v1/station/historical/\"\n       \"klima-v2-1d?parameters=tlmin,tlmax&station_ids=3805\"\n       \"&start=2024-01-01&end=2024-12-31\")\nwith urllib.request.urlopen(URL, timeout=120) as resp:\n    payload = json.load(resp)\nparams = payload[\"features\"][0][\"properties\"][\"parameters\"]\nstamps = [t[:10] for t in payload[\"timestamps\"]]\ntmin = np.array(params[\"tlmin\"][\"data\"], dtype=float)\ntmax = np.array(params[\"tlmax\"][\"data\"], dtype=float)\nwith open(OUT_DIR / \"supervisedlearning_weather.csv\", \"w\") as f:\n    f.write(\"date,tmin,tmax\\n\")\n    for day, lo, hi in zip(stamps, tmin, tmax):\n        f.write(f\"{day},{lo},{hi}\\n\")\ncheck(\"[B-data] 366 labeled days downloaded for 2024\", len(tmin) == 366)\ncheck(\"[B-data] the record matches the archive (Feb 1: -3.8 to 10.4 deg)\",\n      stamps[31] == \"2024-02-01\" and np.isclose(tmin[31], -3.8)\n      and np.isclose(tmax[31], 10.4))\n\nx, y = tmin, tmax                              # feature, label\ntrain = np.arange(len(x)) < 244                # January to August\nval = ~train                                   # September to December\ncheck(\"[B-data] every data point carries a label\", len(x) == len(y))\ncheck(\"[B-data] 244 training days and 122 validation days\",\n      train.sum() == 244 and val.sum() == 122)\n\n\ndef lstsq_fit(xs, ys, degree):\n    \"\"\"ERM over the polynomials of the given degree, squared error loss.\"\"\"\n    A = np.vander(xs, degree + 1)\n    return np.linalg.lstsq(A, ys, rcond=None)[0]\n\n\ndef mse(w, xs, ys):\n    return float(np.mean((np.vander(xs, len(w)) @ w - ys) ** 2))"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-erm]** ERM over the linear model on the training set; check the training error is far below the sample variance of the labels, that the validation error is close to the training error, and that both beat predicting the sample mean of the training labels."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "w_lin = lstsq_fit(x[train], y[train], 1)\nerr_train = mse(w_lin, x[train], y[train])\nerr_val = mse(w_lin, x[val], y[val])\nvar_labels = float(np.var(y[train]))\nerr_mean = float(np.mean((y[train].mean() - y[val]) ** 2))\nprint(f\"  linear model: training error {err_train:.2f}, validation error \"\n      f\"{err_val:.2f}; label variance {var_labels:.2f}, sample-mean \"\n      f\"baseline {err_mean:.2f}\")\ncheck(\"[B-erm] the training error is far below the sample variance of the \"\n      \"labels\", err_train < 0.35 * var_labels)\ncheck(\"[B-erm] the validation error is close to the training error\",\n      err_val < 1.5 * err_train)\ncheck(\"[B-erm] both beat predicting the sample mean of the training labels\",\n      err_val < 0.5 * err_mean)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-overfit]** The same labels, a training set of ten days and a degree-12 polynomial: smaller training error, far larger validation error."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "small = np.zeros(len(x), dtype=bool)\nsmall[np.linspace(0, 243, 10).astype(int)] = True\nxs = (x[small] - x[train].mean()) / x[train].std()     # scaled, for conditioning\nxv = (x[val] - x[train].mean()) / x[train].std()\nw_poly = lstsq_fit(xs, y[small], 12)\npoly_train = mse(w_poly, xs, y[small])\npoly_val = mse(w_poly, xv, y[val])\nw_lin_small = lstsq_fit(xs, y[small], 1)\nprint(f\"  degree-12 polynomial on 10 days: training error {poly_train:.4f}, \"\n      f\"validation error {poly_val:.0f}; linear model on the same 10 days: \"\n      f\"{mse(w_lin_small, xs, y[small]):.2f} and \"\n      f\"{mse(w_lin_small, xv, y[val]):.2f}\")\ncheck(\"[B-overfit] the polynomial has a smaller training error than the \"\n      \"linear model\", poly_train < mse(w_lin_small, xs, y[small]))\ncheck(\"[B-overfit] and a far larger validation error\",\n      poly_val > 10 * mse(w_lin_small, xv, y[val]))"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-classify]** The label space decides the task: the binary label \"frost in the morning\" turns the same records into a classification problem; a threshold rule fitted by ERM with the zero-one loss beats predicting the majority class."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "frost = (tmin < 0.0).astype(int)               # label constructed from tmin\nfeat = tmax                                    # feature: maximum of the day\nthresholds = np.linspace(feat.min(), feat.max(), 400)\nerrs = [np.mean((feat[train] < t).astype(int) != frost[train])\n        for t in thresholds]\nt_hat = float(thresholds[int(np.argmin(errs))])\nacc_val = float(np.mean((feat[val] < t_hat).astype(int) == frost[val]))\nmajority = float(max(frost[train].mean(), 1 - frost[train].mean()))\nacc_major = float(np.mean(frost[val] == int(frost[train].mean() > 0.5)))\nprint(f\"  classification: frost on {100 * frost.mean():.0f}% of the days; \"\n      f\"threshold {t_hat:.1f} deg, validation accuracy {acc_val:.2f} against \"\n      f\"the majority class {acc_major:.2f}\")\ncheck(\"[B-classify] the binary label splits the days into two nonempty \"\n      \"classes\", 0 < frost.sum() < len(frost))\ncheck(\"[B-classify] the threshold rule beats the majority class\",\n      acc_val > acc_major)\ncheck(\"[B-classify] both problems use the same data points, only the label \"\n      \"space differs\", len(feat) == len(y) and majority <= 1.0)\n\n# ---- CSVs for the entry's figure\nnp.savetxt(OUT_DIR / \"supervisedlearning_train.csv\",\n           np.stack([x[train], y[train]], 1), delimiter=\",\",\n           header=\"x,y\", comments=\"\", fmt=\"%.1f\")\nnp.savetxt(OUT_DIR / \"supervisedlearning_val.csv\",\n           np.stack([x[val], y[val]], 1), delimiter=\",\",\n           header=\"x,y\", comments=\"\", fmt=\"%.1f\")\ngrid = np.linspace(x.min() - 1, x.max() + 1, 300)\ngs = (grid - x[train].mean()) / x[train].std()\nwith open(OUT_DIR / \"supervisedlearning_fit.csv\", \"w\") as f:\n    f.write(\"x,linear,poly\\n\")\n    lin = np.vander(grid, 2) @ w_lin\n    pol = np.clip(np.vander(gs, 13) @ w_poly, -15.0, 45.0)\n    for a, b, c in zip(grid, lin, pol):\n        f.write(f\"{a:.2f},{b:.2f},{c:.2f}\\n\")"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-plot]** Preview: the labeled days with the learned hypothesis, and the overfitting polynomial beside it."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 4.2))\nax1.scatter(x[train], y[train], s=12, color=\"black\", label=\"training day\")\nax1.scatter(x[val], y[val], s=18, marker=\"^\", facecolors=\"none\",\n            edgecolors=\"tab:blue\", label=\"validation day\")\nax1.plot(grid, np.vander(grid, 2) @ w_lin, \"r-\", lw=2,\n         label=\"learned hypothesis\")\nax1.set_xlabel(\"feature: morning minimum temperature in deg C\")\nax1.set_ylabel(\"label: maximum temperature of the day in deg C\")\nax1.set_title(\"Each day carries a label; ERM fits a hypothesis to them\")\nax1.legend(frameon=False, fontsize=8)\nax2.scatter(x[small], y[small], s=30, color=\"black\", label=\"the 10 training days\")\nax2.plot(grid, np.vander(grid, 2) @ w_lin_small, \"r-\", lw=2, label=\"linear model\")\nax2.plot(grid, np.clip(np.vander(gs, 13) @ w_poly, -15, 45), \"b--\", lw=1.5,\n         label=\"degree-12 polynomial\")\nax2.set_ylim(-15, 45)\nax2.set_xlabel(\"feature: morning minimum temperature in deg C\")\nax2.set_ylabel(\"label: maximum temperature of the day in deg C\")\nax2.set_title(\"Labels alone do not guarantee generalization\")\nax2.legend(frameon=False, fontsize=8)\nfig.tight_layout()\nfig.savefig(OUT_DIR / \"supervisedlearning.png\", dpi=110)\n\nn_ok = sum(ok for _, ok in report)\nprint(f\"\\n{n_ok}/{len(report)} checks pass\")\nif n_ok != len(report):\n    raise SystemExit(1)"
  }
 ]
}