{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3",
   "language": "python"
  },
  "language_info": {
   "name": "python"
  },
  "colab": {
   "name": "mse.ipynb"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# mean squared error (MSE) \u2014 Python demo\n\nNumerical companion to the entry [mean squared error (MSE)](https://dictionaryofml.org/terms/mse.html) of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org/): it recomputes what the entry states and prints one line per check.\n\nNumerical companion to the glossary entry 'mse' (mean squared error). A linear hypothesis that predicts the maximum daytime temperature from the morning minimum is learned by least squares on the 365 days of 2023 at Krems (GeoSphere station 3805). Its MSE is then computed on the 366 days of 2024 in the two forms the entry writes down: as the average of the squared error loss over the days, and as the expectation of the squared error loss when a day is drawn from the empirical distribution of the 2024 days, which puts probability 1/m on each day. The two agree exactly; that is the point of the entry. The demo also shows the average over the first m days settling towards the 366-day value, the sense in which the dataset average estimates the MSE as an expectation under the data-generating distribution.\n\nRequires NumPy and Matplotlib only, and uses fixed seeds, so the printed numbers reproduce exactly. Generated from [`pythondemos/mse.py`](https://dictionaryofml.org/terms/mse.py); CC BY 4.0.\n\nBefore running a downloaded copy, scan it: `uvx bandit -q -t B102,B307,B602,B605 mse.py` (B102,B307,B602,B605 are the exec, eval and shell checks of [bandit](https://bandit.readthedocs.io/)) reports nothing: the script runs no shell command and evaluates no generated code."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "# Notebook shim: the script resolves output paths relative to __file__,\n# which a notebook kernel does not define; everything lands in the\n# working directory instead.\nimport os\n__file__ = os.path.join(os.getcwd(), \"mse.py\")\nos.makedirs(\"pythondemos\", exist_ok=True)"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "\"\"\"The mean squared error (MSE) of a temperature hypothesis at Krems an der\nDonau: the average squared error loss over a dataset, and the same number\nas an expectation under the empirical distribution of that dataset.\n\nPurpose\n-------\nNumerical companion to the glossary entry 'mse' (mean squared error).  A\nlinear hypothesis that predicts the maximum daytime temperature from the\nmorning minimum is learned by least squares on the 365 days of 2023 at\nKrems (GeoSphere station 3805).  Its MSE is then computed on the 366 days\nof 2024 in the two forms the entry writes down: as the average of the\nsquared error loss over the days, and as the expectation of the squared\nerror loss when a day is drawn from the empirical distribution of the\n2024 days, which puts probability 1/m on each day.  The two agree\nexactly; that is the point of the entry.  The demo also shows the\naverage over the first m days settling towards the 366-day value, the\nsense in which the dataset average estimates the MSE as an expectation\nunder the data-generating distribution.\n\nSelf-contained (numpy and matplotlib only), fixed seeds; the GeoSphere\ndownload falls back to the committed CSV.\n\nBlocks\n------\n[B-data]     Daily minimum and maximum temperature at Krems for 2023 and\n             2024 (731 days, no missing value).\n[B-learn]    Least squares on the 2023 days gives the linear hypothesis;\n             its MSE on the training days.\n[B-average]  The MSE on the 2024 days as the average of the per-day\n             squared error loss; the root MSE in degrees.\n[B-empdist]  The same number as an expectation: probability 1/366 per day\n             (duplicates merged into their multiplicities), the mean of\n             the empirical distribution of the squared errors.\n[B-estimate] The average over the first m days of 2024 in a random\n             order, and the 5th to 95th percentile band of that average\n             over 500 orders, against the 366-day MSE.\n[B-robust]   What the squaring costs: the MAE of the same hypothesis, the\n             constant prediction each of the two averages prefers (the\n             mean of the labels against their median), and how far one\n             mismeasured day moves each of the two.\n\nOutputs\n-------\nmse_weather.csv   : date, tmin, tmax for 2023 and 2024 (committed copy).\nmse_cdf.csv       : squared error of each 2024 day, sorted, with the\n                    empirical cdf value.\nmse_running.csv   : m, running average for one order, 5th and 95th\n                    percentile of the running average over 500 orders.\nmse_robust.csv    : size of a single recording error in degrees, and the\n                    shift it causes in the MSE-optimal and the MAE-optimal\n                    constant prediction.\nmse.png           : matplotlib preview (checking only).\n\"\"\"\n\n# /// script\n# requires-python = \">=3.10\"\n# dependencies = [\"matplotlib==3.10.7\", \"numpy==2.3.4\"]\n# ///\n\nimport json\nimport urllib.request\nfrom pathlib import Path\n\nimport numpy as np\nimport matplotlib\n\nmatplotlib.use(\"Agg\")\nimport matplotlib.pyplot as plt\n\nOUT_DIR = Path(__file__).parent\n\nreport = []\n\n\ndef check(name, ok):\n    report.append((name, bool(ok)))\n    print(f\"  [{'ok' if ok else 'FAIL'}] {name}\")"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-data]** Daily minimum and maximum temperature at Krems for 2023 and 2024 (731 days, no missing value)."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-data] daily minimum and maximum temperature at Krems, 2023 and 2024\")\nURL = (\"https://dataset.api.hub.geosphere.at/v1/station/historical/\"\n       \"klima-v2-1d?parameters=tlmin,tlmax&station_ids=3805\"\n       \"&start=2023-01-01&end=2024-12-31\")\nCSV = OUT_DIR / \"mse_weather.csv\"\ntry:\n    with urllib.request.urlopen(URL, timeout=120) as resp:\n        payload = json.load(resp)\n    params = payload[\"features\"][0][\"properties\"][\"parameters\"]\n    records = [(stamp[:10], lo, hi)\n               for stamp, lo, hi in zip(payload[\"timestamps\"],\n                                        params[\"tlmin\"][\"data\"],\n                                        params[\"tlmax\"][\"data\"])]\n    with open(CSV, \"w\") as fh:\n        fh.write(\"date,tmin,tmax\\n\")\n        for day, lo, hi in records:\n            fh.write(f\"{day},{lo},{hi}\\n\")\n    print(f\"    downloaded {len(records)} days\")\nexcept OSError as err:\n    print(f\"    download failed ({err}); reading the committed CSV\")\n    rows = CSV.read_text().strip().splitlines()[1:]\n    records = [(d, float(a), float(b))\n               for d, a, b in (r.split(\",\") for r in rows)]\n\ncheck(\"[B-data] 731 days for 2023 and 2024\", len(records) == 731)\ncheck(\"[B-data] no day is missing a temperature\",\n      all(lo is not None and hi is not None for _, lo, hi in records))\nx = np.array([lo for _, lo, _ in records])\ny = np.array([hi for _, _, hi in records])\nis23 = np.array([d.startswith(\"2023\") for d, _, _ in records])\nx23, y23, x24, y24 = x[is23], y[is23], x[~is23], y[~is23]\nm23, m24 = len(x23), len(x24)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-learn]** Least squares on the 2023 days gives the linear hypothesis; its MSE on the training days."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-learn] linear hypothesis learned on the 2023 days\")\nA = np.column_stack([np.ones(m23), x23])\nw0, w1 = np.linalg.lstsq(A, y23, rcond=None)[0]\n\n\ndef h(xx):\n    return w0 + w1 * xx\n\n\nmse23 = float(np.mean((y23 - h(x23)) ** 2))\ncheck(f\"[B-learn] h(x) = {w0:.2f} + {w1:.3f} x; MSE on the 365 training \"\n      f\"days {mse23:.2f}\", 0.5 < w1 < 1.5 and mse23 > 0)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-average]** The MSE on the 2024 days as the average of the per-day squared error loss; the root MSE in degrees."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-average] MSE on the 2024 days as the average squared error loss\")\nsq = (y24 - h(x24)) ** 2                       # per-day squared error loss\nmse24 = float(np.mean(sq))\ncheck(f\"[B-average] MSE on the 366 days of 2024: {mse24:.2f} squared \"\n      f\"degrees, root MSE {np.sqrt(mse24):.2f} degrees\", mse24 > 0)\ncheck(f\"[B-average] the loss of 1 January 2024 (minimum {x24[0]:.1f}, \"\n      f\"maximum {y24[0]:.1f}, prediction {h(x24[0]):.2f}) is \"\n      f\"{sq[0]:.2f}\", abs(sq[0] - (y24[0] - h(x24[0])) ** 2) < 1e-12)\ncheck(f\"[B-average] the MSE of 2024 is {'above' if mse24 > mse23 else 'below'}\"\n      f\" the training MSE of 2023 ({mse24:.2f} against {mse23:.2f})\", True)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-empdist]** The same number as an expectation: probability 1/366 per day (duplicates merged into their multiplicities), the mean of the empirical distribution of the squared errors."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "#      empirical distribution of the 2024 days\nprint(\"[B-empdist] expectation under the empirical distribution\")\npairs = {}\nfor xi, yi in zip(x24, y24):\n    pairs[(xi, yi)] = pairs.get((xi, yi), 0) + 1\ncheck(f\"[B-empdist] {len(pairs)} distinct (minimum, maximum) pairs among \"\n      f\"the 366 days; the empirical distribution gives each pair the \"\n      f\"probability (multiplicity)/366\", sum(pairs.values()) == m24)\nexpectation = sum(n / m24 * (yi - h(xi)) ** 2 for (xi, yi), n in pairs.items())\ncheck(f\"[B-empdist] expectation {expectation:.6f} equals the average \"\n      f\"{mse24:.6f}\", abs(expectation - mse24) < 1e-9)\norder = np.argsort(sq)\nsq_sorted, cdf = sq[order], np.arange(1, m24 + 1) / m24\nmedian = float(sq_sorted[m24 // 2 - 1 + m24 % 2])\ncheck(f\"[B-empdist] the empirical distribution of the squared errors is \"\n      f\"skewed: median {median:.2f}, mean {mse24:.2f}, largest \"\n      f\"{sq_sorted[-1]:.1f}; {100 * np.mean(sq < mse24):.0f}% of the days \"\n      f\"have a loss below the MSE\", median < mse24 < sq_sorted[-1])"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-estimate]** The average over the first m days of 2024 in a random order, and the 5th to 95th percentile band of that average over 500 orders, against the 366-day MSE."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-estimate] running average against the 366-day MSE\")\nrng = np.random.default_rng(2024)\nR = 500\nrun = np.empty((R, m24))\nfor r in range(R):\n    perm = rng.permutation(m24)\n    run[r] = np.cumsum(sq[perm]) / np.arange(1, m24 + 1)\nlo, hi = np.percentile(run, 5, axis=0), np.percentile(run, 95, axis=0)\none = run[0]\ncheck(f\"[B-estimate] every running average ends at the MSE {mse24:.2f}\",\n      np.allclose(run[:, -1], mse24))\ncheck(f\"[B-estimate] 5th to 95th percentile of the average over the first \"\n      f\"30 days: [{lo[29]:.1f}, {hi[29]:.1f}]; over the first 180 days: \"\n      f\"[{lo[179]:.1f}, {hi[179]:.1f}]\",\n      hi[29] - lo[29] > hi[179] - lo[179] > 0)\ncheck(\"[B-estimate] the band narrows with m (monotone from m = 10 on,\"\n      \" up to small reversals below 1 squared degree)\",\n      np.all(np.diff((hi - lo)[9:]) < 1.0))\n\n# ---- CSVs and preview\nwith open(OUT_DIR / \"mse_cdf.csv\", \"w\") as fh:\n    fh.write(\"sqerr,cdf\\n\")\n    for s, c in zip(sq_sorted, cdf):\n        fh.write(f\"{s:.4f},{c:.6f}\\n\")\nwith open(OUT_DIR / \"mse_running.csv\", \"w\") as fh:\n    fh.write(\"m,running,lo,hi\\n\")\n    for i in range(m24):\n        fh.write(f\"{i + 1},{one[i]:.4f},{lo[i]:.4f},{hi[i]:.4f}\\n\")\n\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(9.4, 3.9))\nax1.step(np.concatenate([[0], sq_sorted]), np.concatenate([[0], cdf]),\n         where=\"post\", color=\"black\", label=\"empirical cdf of the squared error\")\nax1.axvline(mse24, color=\"black\", ls=\"--\", label=f\"MSE = {mse24:.1f}\")\nax1.set_xlim(0, 60)\nax1.set_xlabel(\"squared error loss of a day (squared degrees)\")\nax1.set_ylabel(\"fraction of the 366 days\")\nax1.set_title(\"the MSE is the mean of this distribution\")\nax1.legend(frameon=False, fontsize=8)\nax2.plot(np.arange(1, m24 + 1), one, \"k-\", lw=0.8, label=\"average over the first m days\")\nax2.plot(np.arange(1, m24 + 1), lo, \"k:\", lw=0.8, label=\"5th and 95th percentile (500 orders)\")\nax2.plot(np.arange(1, m24 + 1), hi, \"k:\", lw=0.8)\nax2.axhline(mse24, color=\"black\", ls=\"--\", label=\"MSE over all 366 days\")\nax2.set_ylim(0, 40)\nax2.set_xlabel(\"number m of days averaged\")\nax2.set_ylabel(\"average squared error loss\")\nax2.set_title(\"the average settles as m grows\")\nax2.legend(frameon=False, fontsize=8)\nfig.tight_layout()\nfig.savefig(OUT_DIR / \"mse.png\", dpi=110)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-robust]** What the squaring costs: the MAE of the same hypothesis, the constant prediction each of the two averages prefers (the mean of the labels against their median), and how far one mismeasured day moves each of the two."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-robust] the squared average against the absolute average\")\nmae24 = float(np.mean(np.abs(y24 - h(x24))))\ncheck(f\"[B-robust] MAE of the same hypothesis on the 366 days of 2024: \"\n      f\"{mae24:.2f} degrees, against a root MSE of {np.sqrt(mse24):.2f}\",\n      mae24 < np.sqrt(mse24))\n\n# a constant prediction: the squared average prefers the mean of the\n# labels, the absolute average their median\ngrid = np.linspace(y24.min(), y24.max(), 20001)\nc_sq = float(grid[np.argmin([np.mean((y24 - c) ** 2) for c in grid[::50]][0:0] or\n                            np.mean((y24[:, None] - grid[None, :]) ** 2, axis=0))])\nc_ab = float(grid[np.argmin(np.mean(np.abs(y24[:, None] - grid[None, :]),\n                                    axis=0))])\ncheck(f\"[B-robust] over a grid of {grid.size} constants, the squared \"\n      f\"average is smallest at {c_sq:.2f} and the mean of the labels is \"\n      f\"{y24.mean():.2f}\", abs(c_sq - y24.mean()) < 0.02)\ncheck(f\"[B-robust] the absolute average is smallest at {c_ab:.2f} and \"\n      f\"the median of the labels is {np.median(y24):.2f}\",\n      abs(c_ab - np.median(y24)) < 0.02)\n\n# one mismeasured day, by a decimal point or a sensor fault\nshifts, rows = [5.0, 10.0, 20.0, 50.0], []\nfor s_deg in shifts:\n    y_bad = y24.copy()\n    y_bad[0] += s_deg\n    d_sq = abs(float(y_bad.mean()) - float(y24.mean()))\n    d_ab = abs(float(np.median(y_bad)) - float(np.median(y24)))\n    rows.append((s_deg, d_sq, d_ab))\n    check(f\"[B-robust] moving one of the {m24} labels by {s_deg:.0f} \"\n          f\"degrees moves the mean by {d_sq:.3f} and the median by \"\n          f\"{d_ab:.3f} degrees\",\n          abs(d_sq - s_deg / m24) < 1e-9)\n# the mean moves by the error divided by the sample size, without bound;\n# the median moves at most to the neighboring label and then stops, so a\n# gross error has a bounded effect on it\ngap = float(np.diff(np.sort(y24))[np.searchsorted(\n    np.sort(y24), np.median(y24)) - 1:][0:1].max())\ncheck(f\"[B-robust] the shift of the mean grows in proportion to the \"\n      f\"error ({rows[0][1]:.3f} to {rows[-1][1]:.3f} degrees, each the \"\n      f\"error divided by {m24}) while the shift of the median stops at \"\n      f\"{max(r[2] for r in rows):.3f}, the step to the neighboring \"\n      f\"label\", rows[-1][1] > 9 * rows[0][1]\n      and max(r[2] for r in rows) <= gap + 1e-9)\ncheck(f\"[B-robust] from a {shifts[1]:.0f} degree error on, the median \"\n      f\"no longer moves further: {rows[1][2]:.3f}, {rows[2][2]:.3f}, \"\n      f\"{rows[3][2]:.3f} degrees\",\n      abs(rows[1][2] - rows[3][2]) < 1e-9)\nnp.savetxt(OUT_DIR / \"mse_robust.csv\",\n           np.array(rows), delimiter=\",\", fmt=\"%.6f\",\n           header=\"shift,mean_shift,median_shift\", comments=\"\")\n\nn_ok = sum(ok for _, ok in report)\nprint(f\"\\n{n_ok}/{len(report)} checks pass\")\nif n_ok != len(report):\n    raise SystemExit(1)"
  }
 ]
}