{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3",
   "language": "python"
  },
  "language_info": {
   "name": "python"
  },
  "colab": {
   "name": "maxlikelihood.ipynb"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# maximum likelihood \u2014 Python demo\n\nNumerical companion to the entry [maximum likelihood](https://dictionaryofml.org/terms/maxlikelihood.html) of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org/): it recomputes what the entry states and prints one line per check.\n\nNumerical companion to the glossary entry 'maxlikelihood'. The 366 daily maximum temperatures of 2024 at Krems (GeoSphere station 3805) are modeled as realizations of i.i.d. Gaussian random variables with unknown mean and variance. Maximum likelihood chooses the two parameters under which the observed year has the largest probability density.\n\nRequires NumPy and Matplotlib only, and uses fixed seeds, so the printed numbers reproduce exactly. Generated from [`pythondemos/maxlikelihood.py`](https://dictionaryofml.org/terms/maxlikelihood.py); CC BY 4.0.\n\nBefore running a downloaded copy, scan it: `uvx bandit -q -t B102,B307,B602,B605 maxlikelihood.py` (B102,B307,B602,B605 are the exec, eval and shell checks of [bandit](https://bandit.readthedocs.io/)) reports nothing: the script runs no shell command and evaluates no generated code."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "# Notebook shim: the script resolves output paths relative to __file__,\n# which a notebook kernel does not define; everything lands in the\n# working directory instead.\nimport os\n__file__ = os.path.join(os.getcwd(), \"maxlikelihood.py\")\nos.makedirs(\"pythondemos\", exist_ok=True)"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "\"\"\"Maximum likelihood for the daily maximum temperature at Krems an der\nDonau: the Gaussian that makes one year of observations most probable.\n\nPurpose\n-------\nNumerical companion to the glossary entry 'maxlikelihood'.  The 366 daily\nmaximum temperatures of 2024 at Krems (GeoSphere station 3805) are modeled\nas realizations of i.i.d. Gaussian random variables with unknown mean and\nvariance.  Maximum likelihood chooses the two parameters under which the\nobserved year has the largest probability density.\n\nThe demo checks the entry's claims: the maximum likelihood estimates of a\nGaussian are the sample mean and the mean squared deviation (closed\nforms, the zero-gradient condition of the log-likelihood); the average\nlog-likelihood as a function of the mean is a concave parabola whose\ncurvature is the negative Fisher information; the negative log-likelihood\nper data point is the average of a loss, an affine function of the\naverage squared error, so maximum likelihood is ERM, the minimization\nof a training error; the variance estimate is biased by the factor\n(m-1)/m; enlarging the hypothesis space to one mean per calendar day lets the\nlikelihood grow without bound while the log-likelihood of the preceding\nyear falls without bound (overfitting), whereas twelve monthly means\nimprove both years.\n\nDeterministic: no random numbers are drawn.  Self-contained: numpy +\nmatplotlib only; the GeoSphere download falls back to the committed CSV.\n\nBlocks\n------\n[B-data]       Download the daily maximum temperature of 2023 and 2024 at\n               Krems (GeoSphere station 3805) or read the committed copy;\n               731 days, no missing value.\n[B-closedform] The Gaussian with the largest likelihood on the 366 days\n               of 2024: mean = sample mean, variance = mean squared\n               deviation; the gradient of the log-likelihood vanishes\n               there and no grid point does better; the variance\n               estimate is (m-1)/m times the sum of squared deviations\n               divided by m-1 (the unbiased estimate of the variance).\n[B-loglik]     The average log-likelihood as a function of the mean, with\n               the variance fixed at its estimate: a concave parabola\n               with vertex at the sample mean and second derivative\n               -1/variance, the negative Fisher information; the same\n               curve for the 365 days of 2023 peaks at the 2023 mean.\n[B-erm]        The negative log-likelihood per day is the average of a\n               loss; for a fixed variance it is an affine function of the\n               average squared error, so both have the same minimizer.\n[B-hypospace]  Three hypothesis spaces: one Gaussian, twelve monthly\n               means with a common variance, one mean per calendar day\n               with a common variance.  Average log-likelihood of the\n               training year 2024 and of the held-out year 2023: the\n               monthly means improve both, the daily means drive the\n               training value above every bound and the held-out value\n               below every bound as the variance shrinks to zero.\n[B-plot]       Histogram and Gaussian pdf, the two log-likelihood\n               curves, and the preview figure.\n\nOutputs\n-------\nmaxlikelihood_weather.csv : date, tmax for 2023 and 2024 (committed copy).\nmaxlikelihood_hist.csv    : t, freq -- histogram of the 2024 maxima,\n                            bin edges (ybar interval), density scale.\nmaxlikelihood_pdf.csv     : t, pdf -- the maximum likelihood Gaussian.\nmaxlikelihood_loglik.csv  : mu, ll2024, ll2023 -- average log-likelihood\n                            against the mean, variance fixed at the 2024\n                            estimate.\nmaxlikelihood.png         : matplotlib preview (checking only).\n\"\"\"\n\n# /// script\n# requires-python = \">=3.10\"\n# dependencies = [\"matplotlib==3.10.7\", \"numpy==2.3.4\"]\n# ///\n\nimport json\nimport urllib.request\nfrom pathlib import Path\n\nimport numpy as np\nimport matplotlib\n\nmatplotlib.use(\"Agg\")\nimport matplotlib.pyplot as plt\n\nOUT_DIR = Path(__file__).parent\n\nreport = []\n\n\ndef check(name, ok):\n    report.append((name, bool(ok)))\n    print(f\"  [{'ok' if ok else 'FAIL'}] {name}\")"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-data]** Download the daily maximum temperature of 2023 and 2024 at Krems (GeoSphere station 3805) or read the committed copy; 731 days, no missing value."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-data] daily maximum temperature at Krems, 2023 and 2024\")\nURL = (\"https://dataset.api.hub.geosphere.at/v1/station/historical/\"\n       \"klima-v2-1d?parameters=tlmax&station_ids=3805\"\n       \"&start=2023-01-01&end=2024-12-31\")\nCSV = OUT_DIR / \"maxlikelihood_weather.csv\"\ntry:\n    with urllib.request.urlopen(URL, timeout=120) as resp:\n        payload = json.load(resp)\n    values = payload[\"features\"][0][\"properties\"][\"parameters\"][\"tlmax\"]\n    records = [(stamp[:10], hi)\n               for stamp, hi in zip(payload[\"timestamps\"], values[\"data\"])]\n    with open(CSV, \"w\") as fh:\n        fh.write(\"date,tmax\\n\")\n        for day, hi in records:\n            fh.write(f\"{day},{hi}\\n\")\n    print(f\"    downloaded {len(records)} days\")\nexcept OSError as err:\n    print(f\"    download failed ({err}); reading the committed CSV\")\n    rows = CSV.read_text().strip().splitlines()[1:]\n    records = [(d, float(v)) for d, v in (r.split(\",\") for r in rows)]\n\ncheck(\"[B-data] 731 days for 2023 and 2024\", len(records) == 731)\ncheck(\"[B-data] no day is missing its maximum temperature\",\n      all(hi is not None for _, hi in records))\ndates = np.array([d for d, _ in records])\ntmax = np.array([hi for _, hi in records], dtype=float)\nis24 = np.array([d.startswith(\"2024\") for d in dates])\nx = tmax[is24]            # the training year: 366 realizations\nx23 = tmax[~is24]         # the held-out year: 365 realizations\nm = len(x)\ncheck(f\"[B-data] {m} days in 2024 and {len(x23)} days in 2023\",\n      m == 366 and len(x23) == 365)\n\n\ndef avg_loglik(data, mu, var):\n    \"\"\"Average log-likelihood of the data under the Gaussian N(mu, var):\n    the mean of ln p(x^(r)) over the data points.\"\"\"\n    return float(np.mean(-0.5 * np.log(2 * np.pi * var)\n                         - (data - mu) ** 2 / (2 * var)))\n\n\ndef gaussian_pdf(t, mu, var):\n    return np.exp(-(t - mu) ** 2 / (2 * var)) / np.sqrt(2 * np.pi * var)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-closedform]** The Gaussian with the largest likelihood on the 366 days of 2024: mean = sample mean, variance = mean squared deviation; the gradient of the log-likelihood vanishes there and no grid point does better; the variance estimate is (m-1)/m times the sum of squared deviations divided by m-1 (the unbiased estimate of the variance)."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-closedform] maximum likelihood Gaussian for the 366 days of 2024\")\nmu_hat = float(np.mean(x))\nvar_hat = float(np.mean((x - mu_hat) ** 2))\ncheck(f\"[B-closedform] mean estimate {mu_hat:.1f} C, variance estimate \"\n      f\"{var_hat:.1f} C^2 (standard deviation {np.sqrt(var_hat):.1f} C)\",\n      10 < mu_hat < 25 and 50 < var_hat < 120)\n# the zero-gradient condition of the average log-likelihood at the estimate\ngrad_mu = float(np.mean(x - mu_hat) / var_hat)\ngrad_var = float(-0.5 / var_hat + np.mean((x - mu_hat) ** 2) / (2 * var_hat ** 2))\ncheck(\"[B-closedform] the gradient of the average log-likelihood vanishes \"\n      f\"at the estimate ({grad_mu:.1e}, {grad_var:.1e})\",\n      abs(grad_mu) < 1e-12 and abs(grad_var) < 1e-12)\nll_hat = avg_loglik(x, mu_hat, var_hat)\nMU_GRID = np.linspace(mu_hat - 10, mu_hat + 10, 401)\nVAR_GRID = np.linspace(0.3 * var_hat, 3 * var_hat, 271)\ngrid_best = max(avg_loglik(x, mu, var) for mu in MU_GRID for var in VAR_GRID)\ncheck(f\"[B-closedform] no (mean, variance) on a 401 x 271 grid beats the \"\n      f\"closed form (average log-likelihood {ll_hat:.3f})\",\n      grid_best <= ll_hat + 1e-12)\ncheck(\"[B-closedform] the average log-likelihood at the estimate is \"\n      \"-(ln(2 pi var) + 1)/2\",\n      abs(ll_hat + 0.5 * (np.log(2 * np.pi * var_hat) + 1)) < 1e-12)\ns2_unbiased = float(np.var(x, ddof=1))\ncheck(f\"[B-closedform] the variance estimate is (m-1)/m = {(m - 1) / m:.4f} \"\n      f\"times the sum of squared deviations over m-1, {s2_unbiased:.1f}\",\n      abs(var_hat - (m - 1) / m * s2_unbiased) < 1e-9)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-loglik]** The average log-likelihood as a function of the mean, with the variance fixed at its estimate: a concave parabola with vertex at the sample mean and second derivative -1/variance, the negative Fisher information; the same curve for the 365 days of 2023 peaks at the 2023 mean."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-loglik] average log-likelihood against the mean, variance fixed\")\nll24 = np.array([avg_loglik(x, mu, var_hat) for mu in MU_GRID])\nll23 = np.array([avg_loglik(x23, mu, var_hat) for mu in MU_GRID])\ncheck(f\"[B-loglik] the 2024 curve peaks at the sample mean \"\n      f\"{MU_GRID[np.argmax(ll24)]:.2f} C\",\n      abs(MU_GRID[np.argmax(ll24)] - mu_hat) < 0.05)\n# the parabola: ll(mu) = ll_hat - (mu - mu_hat)^2 / (2 var)\nparabola = ll_hat - (MU_GRID - mu_hat) ** 2 / (2 * var_hat)\ncheck(\"[B-loglik] the curve is the parabola ll_hat - (mu - mean)^2/(2 var)\",\n      np.allclose(ll24, parabola, atol=1e-12))\ncheck(f\"[B-loglik] its second derivative is -1/var = {-1 / var_hat:.4f}, the \"\n      f\"negative Fisher information of the mean per data point\",\n      abs(np.polyfit(MU_GRID, ll24, 2)[0] * 2 + 1 / var_hat) < 1e-9)\nse_mean = float(np.sqrt(var_hat / m))\ncheck(f\"[B-loglik] standard deviation of the mean estimate sqrt(var/m) = \"\n      f\"{se_mean:.2f} C\", 0.3 < se_mean < 0.7)\nmu23 = float(np.mean(x23))\ncheck(f\"[B-loglik] the 2023 curve peaks at the 2023 mean {mu23:.1f} C, \"\n      f\"{abs(mu23 - mu_hat):.1f} C away from the 2024 mean\",\n      abs(MU_GRID[np.argmax(ll23)] - mu23) < 0.05)\ncheck(\"[B-loglik] the 2024 estimate scores lower on 2023 than the 2023 \"\n      f\"mean does: {avg_loglik(x23, mu_hat, var_hat):.4f} < \"\n      f\"{avg_loglik(x23, mu23, var_hat):.4f}\",\n      avg_loglik(x23, mu_hat, var_hat) < avg_loglik(x23, mu23, var_hat))"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-erm]** The negative log-likelihood per day is the average of a loss; for a fixed variance it is an affine function of the average squared error, so both have the same minimizer."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-erm] negative log-likelihood per day as an average loss\")\nloss = lambda xr, mu, var: 0.5 * np.log(2 * np.pi * var) + (xr - mu) ** 2 / (2 * var)\nper_day = np.array([loss(xr, mu_hat, var_hat) for xr in x])\ncheck(f\"[B-erm] average of the 366 per-day losses {per_day.mean():.4f} equals \"\n      f\"the negative average log-likelihood {-ll_hat:.4f}\",\n      abs(per_day.mean() + ll_hat) < 1e-12)\nmse = np.array([np.mean((x - mu) ** 2) for mu in MU_GRID])\ncheck(\"[B-erm] for a fixed variance the negative log-likelihood is \"\n      \"ln(2 pi var)/2 + (average squared error)/(2 var) on the whole grid\",\n      np.allclose(-ll24, 0.5 * np.log(2 * np.pi * var_hat) + mse / (2 * var_hat)))\ncheck(\"[B-erm] the mean with the smallest average squared error is the \"\n      \"maximum likelihood mean\",\n      abs(MU_GRID[np.argmin(mse)] - mu_hat) < 0.05)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-hypospace]** Three hypothesis spaces: one Gaussian, twelve monthly means with a common variance, one mean per calendar day with a common variance. Average log-likelihood of the training year 2024 and of the held-out year 2023: the monthly means improve both, the daily means drive the training value above every bound and the held-out value below every bound as the variance shrinks to zero."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-hypospace] one Gaussian, monthly means, daily means\")\nmonth24 = np.array([int(d[5:7]) for d in dates[is24]])\nmonth23 = np.array([int(d[5:7]) for d in dates[~is24]])\nmonthday24 = np.array([d[5:] for d in dates[is24]])\nmonthday23 = np.array([d[5:] for d in dates[~is24]])\n\n\ndef avg_loglik_means(data, means, var):\n    return float(np.mean(-0.5 * np.log(2 * np.pi * var)\n                         - (data - means) ** 2 / (2 * var)))\n\n\n# (A) one Gaussian, learned on 2024\ntrain_A = ll_hat\ntest_A = avg_loglik(x23, mu_hat, var_hat)\n# (B) twelve monthly means and one common variance, learned on 2024\nmu_month = {k: float(np.mean(x[month24 == k])) for k in range(1, 13)}\nmeans24_B = np.array([mu_month[k] for k in month24])\nvar_B = float(np.mean((x - means24_B) ** 2))\nmeans23_B = np.array([mu_month[k] for k in month23])\ntrain_B = avg_loglik_means(x, means24_B, var_B)\ntest_B = avg_loglik_means(x23, means23_B, var_B)\ncheck(f\"[B-hypospace] one Gaussian: average log-likelihood {train_A:.2f} on \"\n      f\"2024 (training year), {test_A:.2f} on 2023 (held-out year)\",\n      abs(train_A - test_A) < 0.1)\ncheck(f\"[B-hypospace] twelve monthly means (variance {var_B:.1f}): \"\n      f\"{train_B:.2f} on 2024, {test_B:.2f} on 2023, better on both years\",\n      train_B > train_A and test_B > test_A)\n# (C) one mean per calendar day and one common variance: the maximum\n# likelihood means are the observations themselves and the variance 0\nmu_day = dict(zip(monthday24, x))\nmeans24_C = np.array([mu_day[k] for k in monthday24])\nmeans23_C = np.array([mu_day[k] for k in monthday23])\nvar_C = float(np.mean((x - means24_C) ** 2))\ncheck(\"[B-hypospace] daily means: every mean equals its observation, the \"\n      f\"variance estimate is {var_C:.1f}\", var_C == 0.0)\nsigmas = (1.0, 0.1, 0.01)\ntrain_C = [avg_loglik_means(x, means24_C, s ** 2) for s in sigmas]\ntest_C = [avg_loglik_means(x23, means23_C, s ** 2) for s in sigmas]\ncheck(\"[B-hypospace] daily means, standard deviation 1, 0.1, 0.01: on 2024 \"\n      + \", \".join(f\"{v:.2f}\" for v in train_C)\n      + \" (rises by ln 10 per step, no upper bound)\",\n      all(b > a for a, b in zip(train_C, train_C[1:]))\n      and all(abs((b - a) - np.log(10)) < 1e-9\n              for a, b in zip(train_C, train_C[1:])))\ncheck(\"[B-hypospace] daily means on 2023: \"\n      + \", \".join(f\"{v:.0f}\" for v in test_C)\n      + \" (falls without bound, below the single Gaussian)\",\n      all(b < a for a, b in zip(test_C, test_C[1:])) and test_C[0] < test_A)\ncheck(f\"[B-hypospace] mean squared difference between the same calendar \"\n      f\"days of 2023 and 2024: {np.mean((x23 - means23_C) ** 2):.1f} C^2\",\n      np.mean((x23 - means23_C) ** 2) > var_B)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-plot]** Histogram and Gaussian pdf, the two log-likelihood curves, and the preview figure."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-plot] histogram, Gaussian pdf, log-likelihood curves\")\nedges = np.arange(-6.0, 42.0 + 1e-9, 2.0)\ncounts, edges = np.histogram(x, bins=edges, density=True)\ncheck(f\"[B-plot] {len(counts)} bins of 2 C from {edges[0]:.0f} to \"\n      f\"{edges[-1]:.0f} C hold all {m} days\",\n      x.min() >= edges[0] and x.max() < edges[-1]\n      and abs(np.sum(counts) * 2.0 - 1.0) < 1e-12)\nwith open(OUT_DIR / \"maxlikelihood_hist.csv\", \"w\") as fh:\n    fh.write(\"t,freq\\n\")\n    for left, c in zip(edges[:-1], counts):\n        fh.write(f\"{left:.1f},{c:.5f}\\n\")\n    fh.write(f\"{edges[-1]:.1f},{counts[-1]:.5f}\\n\")   # closes the last bar\ngrid = np.linspace(-8, 44, 261)\nwith open(OUT_DIR / \"maxlikelihood_pdf.csv\", \"w\") as fh:\n    fh.write(\"t,pdf\\n\")\n    for t, p in zip(grid, gaussian_pdf(grid, mu_hat, var_hat)):\n        fh.write(f\"{t:.2f},{p:.6f}\\n\")\nwith open(OUT_DIR / \"maxlikelihood_loglik.csv\", \"w\") as fh:\n    fh.write(\"mu,ll2024,ll2023\\n\")\n    for mu, a, b in zip(MU_GRID[::4], ll24[::4], ll23[::4]):\n        fh.write(f\"{mu:.3f},{a:.5f},{b:.5f}\\n\")\n\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(9.4, 3.8))\nax1.bar(edges[:-1], counts, width=2.0, align=\"edge\", color=\"0.8\",\n        edgecolor=\"0.5\", label=\"histogram of the 366 days of 2024\")\nax1.plot(grid, gaussian_pdf(grid, mu_hat, var_hat), \"k-\", lw=1.6,\n         label=f\"Gaussian pdf, mean {mu_hat:.1f}, s.d. {np.sqrt(var_hat):.1f}\")\nax1.set_xlabel(\"daily maximum temperature (\u00b0C)\")\nax1.set_ylabel(\"relative frequency / pdf\")\nax1.set_title(\"the Gaussian that makes 2024 most probable\")\nax1.legend(frameon=False, fontsize=8)\nax2.plot(MU_GRID, ll24, \"k-\", lw=1.6, label=\"2024 (training year)\")\nax2.plot(MU_GRID, ll23, \"k--\", lw=1.4, label=\"2023 (held-out year)\")\nax2.axvline(mu_hat, color=\"0.5\", ls=\":\", lw=1.2,\n            label=f\"sample mean of 2024 ({mu_hat:.1f} \u00b0C)\")\nax2.set_xlabel(\"mean of the Gaussian (\u00b0C), variance fixed at its estimate\")\nax2.set_ylabel(\"average log-likelihood\")\nax2.set_title(\"log-likelihood as a function of the mean\")\nax2.legend(frameon=False, fontsize=8)\nfig.tight_layout()\nfig.savefig(OUT_DIR / \"maxlikelihood.png\", dpi=110)\n\nn_ok = sum(ok for _, ok in report)\nprint(f\"\\n{n_ok}/{len(report)} checks pass\")\nif n_ok != len(report):\n    raise SystemExit(1)"
  }
 ]
}