{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3",
   "language": "python"
  },
  "language_info": {
   "name": "python"
  },
  "colab": {
   "name": "lda.ipynb"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# linear discriminant analysis (LDA) \u2014 Python demo\n\nNumerical companion to the entry [linear discriminant analysis (LDA)](https://dictionaryofml.org/terms/lda.html) of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org/): it recomputes what the entry states and prints one line per check.\n\nNumerical companion to the glossary entry 'lda'. A day at Krems (GeoSphere station 3805) is a data point with the feature vector (morning minimum temperature, maximum daytime temperature) and the label +1 if precipitation was recorded on that day and -1 otherwise. The days of 2020 to 2023 are the training set and the days of 2024 the test set.\n\nRequires NumPy and Matplotlib only, and uses fixed seeds, so the printed numbers reproduce exactly. Generated from [`pythondemos/lda.py`](https://dictionaryofml.org/terms/lda.py); CC BY 4.0.\n\nBefore running a downloaded copy, scan it: `uvx bandit -q -t B102,B307,B602,B605 lda.py` (B102,B307,B602,B605 are the exec, eval and shell checks of [bandit](https://bandit.readthedocs.io/)) reports nothing: the script runs no shell command and evaluates no generated code."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "# Notebook shim: the script resolves output paths relative to __file__,\n# which a notebook kernel does not define; everything lands in the\n# working directory instead.\nimport os\n__file__ = os.path.join(os.getcwd(), \"lda.py\")\nos.makedirs(\"pythondemos\", exist_ok=True)"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "\"\"\"Linear discriminant analysis (LDA) against principal component analysis\n(PCA) on the days at Krems an der Donau: one direction chosen without the\nlabels, one chosen with them.\n\nPurpose\n-------\nNumerical companion to the glossary entry 'lda'.  A day at Krems\n(GeoSphere station 3805) is a data point with the feature vector\n(morning minimum temperature, maximum daytime temperature) and the label\n+1 if precipitation was recorded on that day and -1 otherwise.  The days\nof 2020 to 2023 are the training set and the days of 2024 the test set.\n\nThe demo checks the entry's claims.  The sample covariance matrix of the\ntwo features is the sum of the within-class and the between-class\nscatter matrix, which is the identity that separates the two methods:\nPCA maximizes the quadratic form of the sum over unit-norm directions,\nLDA the ratio of the two parts.  The first principal direction carries\n93 percent of the sample variance and almost none of the label\ninformation, so the Fisher criterion is three hundred times larger along\nthe LDA direction than along it, and the two directions are 89 degrees\napart; a sweep over all directions confirms that the LDA direction\nattains the maximum.  A threshold on the single\nLDA feature classifies the days of 2024 better than a threshold on the\nsingle PCA feature, which does not beat predicting the majority label.\n\nSelf-contained (numpy and matplotlib only), no random numbers; the\nGeoSphere download falls back to the committed CSV.\n\nBlocks\n------\n[B-data]     Daily minimum and maximum temperature and precipitation at\n             Krems for 2020 to 2024 (1827 days, no missing value); the\n             label is +1 on a day with precipitation.\n[B-pca]      The sample covariance matrix of the two features on the\n             training days, its eigenvalues and the first principal\n             direction.\n[B-scatter]  The within-class and the between-class scatter matrix, and\n             the check that they sum to the sample covariance matrix.\n[B-lda]      The LDA direction as the solution of a linear system, the\n             Fisher criterion along it and along the first principal\n             direction, and a sweep over all directions confirming the\n             maximum.\n[B-classify] A threshold on each of the two single features, chosen on\n             the training days, and the accuracy it reaches on the days\n             of 2024.\n[B-plot]     Twenty of the 366 days of 2024 for the figure, the two\n             directions as arrows, and the Fisher criterion against the\n             angle of the direction.\n\nOutputs\n-------\nlda_weather.csv    : date, tmin, tmax, rr for 2020 to 2024 (committed\n                     copy, read when the download fails).\nlda_wet_plot.csv   : the 10 drawn days of 2024 with precipitation.\nlda_dry_plot.csv   : the 10 drawn days of 2024 without precipitation.\nlda_axis_pca.csv   : the two endpoints of the arrow along the first\n                     principal direction.\nlda_axis_lda.csv   : the two endpoints of the arrow along the LDA\n                     direction.\nlda_fisher.csv     : angle in degrees, Fisher criterion of the direction\n                     at that angle.\nlda.png            : matplotlib preview (checking only).\n\"\"\"\n\n# /// script\n# requires-python = \">=3.10\"\n# dependencies = [\"matplotlib==3.10.7\", \"numpy==2.3.4\"]\n# ///\n\nimport json\nimport urllib.request\nfrom pathlib import Path\n\nimport numpy as np\nimport matplotlib\n\nmatplotlib.use(\"Agg\")\nimport matplotlib.pyplot as plt\n\nOUT_DIR = Path(__file__).parent\n\nreport = []\n\n\ndef check(name, ok):\n    report.append((name, bool(ok)))\n    print(f\"  [{'ok' if ok else 'FAIL'}] {name}\")\n\n\nWET = 0.1          # millimetres of precipitation that count as a wet day\n                   # (the archive writes -1 for \"no precipitation\")"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-data]** Daily minimum and maximum temperature and precipitation at Krems for 2020 to 2024 (1827 days, no missing value); the label is +1 on a day with precipitation."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-data] daily temperatures and precipitation at Krems, 2020 to 2024\")\nURL = (\"https://dataset.api.hub.geosphere.at/v1/station/historical/\"\n       \"klima-v2-1d?parameters=tlmin,tlmax,rr&station_ids=3805\"\n       \"&start=2020-01-01&end=2024-12-31\")\nCSV = OUT_DIR / \"lda_weather.csv\"\ntry:\n    with urllib.request.urlopen(URL, timeout=180) as resp:\n        payload = json.load(resp)\n    par = payload[\"features\"][0][\"properties\"][\"parameters\"]\n    records = [(stamp[:10], lo, hi, rain)\n               for stamp, lo, hi, rain in zip(payload[\"timestamps\"],\n                                              par[\"tlmin\"][\"data\"],\n                                              par[\"tlmax\"][\"data\"],\n                                              par[\"rr\"][\"data\"])]\n    with open(CSV, \"w\") as fh:\n        fh.write(\"date,tmin,tmax,rr\\n\")\n        for day, lo, hi, rain in records:\n            fh.write(f\"{day},{lo},{hi},{rain}\\n\")\n    print(f\"    downloaded {len(records)} days\")\nexcept OSError as err:\n    print(f\"    download failed ({err}); reading the committed CSV\")\n    rows = CSV.read_text().strip().splitlines()[1:]\n    records = [(d, float(a), float(b), float(c))\n               for d, a, b, c in (r.split(\",\") for r in rows)]\n\ncheck(\"[B-data] 1827 days for 2020 to 2024\", len(records) == 1827)\ncheck(\"[B-data] no day is missing a measurement\",\n      all(v is not None for rec in records for v in rec[1:]))\nX = np.array([[lo, hi] for _, lo, hi, _ in records])\nrr = np.array([rain for _, _, _, rain in records])\ny = np.where(rr >= WET, 1.0, -1.0)\nyear = np.array([int(d[:4]) for d, _, _, _ in records])\ntr, te = year <= 2023, year == 2024\ncheck(f\"[B-data] {int((y[tr] > 0).sum())} of the {int(tr.sum())} training \"\n      f\"days and {int((y[te] > 0).sum())} of the {int(te.sum())} days of \"\n      f\"2024 carry precipitation\",\n      tr.sum() == 1461 and te.sum() == 366)\nXtr, ytr, Xte, yte = X[tr], y[tr], X[te], y[te]\nm = len(Xtr)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-pca]** The sample covariance matrix of the two features on the training days, its eigenvalues and the first principal direction."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-pca] first principal direction of the two features\")\nmean = Xtr.mean(axis=0)\nXc = Xtr - mean\nQ = Xc.T @ Xc / m\nlam, U = np.linalg.eigh(Q)              # ascending eigenvalues\nlam, U = lam[::-1], U[:, ::-1]\nu1 = U[:, 0] * np.sign(U[:, 0].sum())   # point it towards warmer days\ncheck(f\"[B-pca] sample mean ({mean[0]:.2f}, {mean[1]:.2f}) degrees; \"\n      f\"eigenvalues {lam[0]:.1f} and {lam[1]:.1f} squared degrees\",\n      lam[0] > lam[1] > 0)\ncheck(f\"[B-pca] the first principal direction ({u1[0]:.3f}, {u1[1]:.3f}) \"\n      f\"carries {100 * lam[0] / lam.sum():.0f} percent of the sample \"\n      f\"variance\", lam[0] / lam.sum() > 0.9)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-scatter]** The within-class and the between-class scatter matrix, and the check that they sum to the sample covariance matrix."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-scatter] the two scatter matrices and their sum\")\nmu = {c: Xtr[ytr == c].mean(axis=0) for c in (-1.0, 1.0)}\nfrac = {c: (ytr == c).mean() for c in (-1.0, 1.0)}\nS = sum(((Xtr[ytr == c] - mu[c]).T @ (Xtr[ytr == c] - mu[c])) / m\n        for c in (-1.0, 1.0))\nB = sum(frac[c] * np.outer(mu[c] - mean, mu[c] - mean) for c in (-1.0, 1.0))\ncheck(f\"[B-scatter] class means: dry ({mu[-1.0][0]:.2f}, \"\n      f\"{mu[-1.0][1]:.2f}), wet ({mu[1.0][0]:.2f}, {mu[1.0][1]:.2f}) \"\n      f\"degrees\", mu[1.0][0] > mu[-1.0][0])\ncheck(\"[B-scatter] the within-class and the between-class scatter matrix \"\n      \"sum to the sample covariance matrix\", np.allclose(S + B, Q))\ncheck(f\"[B-scatter] the between-class scatter has rank 1 and trace \"\n      f\"{np.trace(B):.3f}, against {np.trace(S):.1f} for the within-class \"\n      f\"scatter\", np.linalg.matrix_rank(B) == 1)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-lda]** The LDA direction as the solution of a linear system, the Fisher criterion along it and along the first principal direction, and a sweep over all directions confirming the maximum."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-lda] the LDA direction and the Fisher criterion\")\nw = np.linalg.solve(S, mu[1.0] - mu[-1.0])\nw = w / np.linalg.norm(w)\n\n\ndef fisher(d):\n    return float((d @ B @ d) / (d @ S @ d))\n\n\ndiff = mu[1.0] - mu[-1.0]\ncheck(f\"[B-lda] the LDA direction ({w[0]:.3f}, {w[1]:.3f}) is the \"\n      f\"direction that the within-class scatter maps onto the difference \"\n      f\"of the two class means\",\n      abs((S @ w)[0] * diff[1] - (S @ w)[1] * diff[0]) < 1e-12\n      and (S @ w) @ diff > 0)\ncheck(f\"[B-lda] Fisher criterion {fisher(w):.4f} along the LDA direction \"\n      f\"against {fisher(u1):.5f} along the first principal direction, a \"\n      f\"factor of {fisher(w) / fisher(u1):.0f}\", fisher(w) > 50 * fisher(u1))\nang = np.arange(0.0, 180.0, 0.05)\nJ = np.array([fisher(np.array([np.cos(np.deg2rad(a)), np.sin(np.deg2rad(a))]))\n              for a in ang])\nbest = ang[int(np.argmax(J))]\nw_ang = np.rad2deg(np.arctan2(w[1], w[0])) % 180.0\ncheck(f\"[B-lda] a sweep over all directions peaks at {best:.2f} degrees, \"\n      f\"the angle {w_ang:.2f} degrees of the LDA direction\",\n      abs(best - w_ang) < 0.1 and abs(J.max() - fisher(w)) < 1e-6)\ncheck(f\"[B-lda] the LDA direction weighs the two features with opposite \"\n      f\"signs, so its feature falls with the daily temperature range; \"\n      f\"its angle to the first principal direction is \"\n      f\"{np.rad2deg(np.arccos(abs(w @ u1))):.0f} degrees\", w[0] * w[1] < 0)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-classify]** A threshold on each of the two single features, chosen on the training days, and the accuracy it reaches on the days of 2024."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-classify] a threshold on each of the two single features\")\n\n\ndef best_threshold(scores, labels):\n    cand = np.unique(scores)\n    cand = np.concatenate([[cand[0] - 1.0], (cand[:-1] + cand[1:]) / 2])\n    acc = [max(((np.where(scores > t, 1.0, -1.0) == labels).mean(),\n                ((np.where(scores > t, -1.0, 1.0) == labels).mean()))\n               ) for t in cand]\n    i = int(np.argmax(acc))\n    hi = (np.where(cand[i] < scores, 1.0, -1.0) == labels).mean() >= acc[i]\n    return float(cand[i]), bool(hi), float(acc[i])\n\n\nbase = max((yte == 1.0).mean(), (yte == -1.0).mean())\nres = {}\nfor tag, d in ((\"LDA\", w), (\"PCA\", u1)):\n    t, hi, acc_tr = best_threshold(Xtr @ d, ytr)\n    pred = np.where(Xte @ d > t, 1.0, -1.0) if hi else \\\n        np.where(Xte @ d > t, -1.0, 1.0)\n    res[tag] = float((pred == yte).mean())\n    check(f\"[B-classify] threshold {t:.2f} on the {tag} feature: accuracy \"\n          f\"{acc_tr:.3f} on the training days, {res[tag]:.3f} on the days \"\n          f\"of 2024\", 0.0 < res[tag] < 1.0)\ncheck(f\"[B-classify] predicting the majority label reaches {base:.3f} on \"\n      f\"the days of 2024, so the PCA feature gains \"\n      f\"{100 * (res['PCA'] - base):+.1f} and the LDA feature \"\n      f\"{100 * (res['LDA'] - base):+.1f} percentage points\",\n      res[\"LDA\"] - base > 0.02 > res[\"PCA\"] - base)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-plot]** Twenty of the 366 days of 2024 for the figure, the two directions as arrows, and the Fisher criterion against the angle of the direction."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-plot] twenty drawn days, the two arrows, the criterion curve\")\nwet_idx = np.where(yte > 0)[0]\ndry_idx = np.where(yte < 0)[0]\nstep = {\"wet\": len(wet_idx) // 10, \"dry\": len(dry_idx) // 10}\ndraw = {\"wet\": wet_idx[::step[\"wet\"]][:10], \"dry\": dry_idx[::step[\"dry\"]][:10]}\ncheck(f\"[B-plot] one of every {step['wet']} of the {len(wet_idx)} wet days \"\n      f\"and one of every {step['dry']} of the {len(dry_idx)} dry days of \"\n      f\"2024 are drawn, ten of each\",\n      len(draw[\"wet\"]) == len(draw[\"dry\"]) == 10)\nfor tag, idx in draw.items():\n    with open(OUT_DIR / f\"lda_{tag}_plot.csv\", \"w\") as fh:\n        fh.write(\"x1,x2\\n\")\n        for i in idx:\n            fh.write(f\"{Xte[i, 0]:.1f},{Xte[i, 1]:.1f}\\n\")\nLEN = 9.0                               # arrow length in degrees\nfor tag, d in ((\"pca\", u1), (\"lda\", w)):\n    with open(OUT_DIR / f\"lda_axis_{tag}.csv\", \"w\") as fh:\n        fh.write(\"x1,x2\\n\")\n        fh.write(f\"{mean[0]:.3f},{mean[1]:.3f}\\n\")\n        fh.write(f\"{mean[0] + LEN * d[0]:.3f},{mean[1] + LEN * d[1]:.3f}\\n\")\n    print(f\"    arrow {tag}: tip at ({mean[0] + LEN * d[0]:.2f}, \"\n          f\"{mean[1] + LEN * d[1]:.2f})\")\nwith open(OUT_DIR / \"lda_fisher.csv\", \"w\") as fh:\n    fh.write(\"angle,J\\n\")\n    for a, j in zip(ang[::20], J[::20]):\n        fh.write(f\"{a:.2f},{j:.5f}\\n\")\nprint(f\"    criterion curve: {len(ang[::20])} angles, maximum \"\n      f\"{J.max():.4f} at {best:.2f} degrees, minimum {J.min():.5f}\")\n\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(9.6, 4.0))\nax1.plot(Xte[draw[\"dry\"], 0], Xte[draw[\"dry\"], 1], \"s\", mfc=\"none\",\n         mec=\"black\", label=\"day without precipitation\")\nax1.plot(Xte[draw[\"wet\"], 0], Xte[draw[\"wet\"], 1], \"o\", color=\"black\",\n         label=\"day with precipitation\")\nax1.annotate(\"\", xy=mean + LEN * u1, xytext=mean,\n             arrowprops=dict(arrowstyle=\"->\", color=\"black\", lw=1.6))\nax1.annotate(\"\", xy=mean + LEN * w, xytext=mean,\n             arrowprops=dict(arrowstyle=\"->\", color=\"black\", lw=1.6, ls=\"--\"))\nax1.text(*(mean + LEN * u1), \" PCA\", va=\"bottom\", fontsize=8)\nax1.text(*(mean + LEN * w), \" LDA\", va=\"top\", fontsize=8)\nax1.set_xlabel(\"morning minimum temperature in degrees Celsius\")\nax1.set_ylabel(\"maximum daytime temperature in degrees Celsius\")\nax1.set_title(\"ten wet and ten dry days of 2024\")\nax1.legend(frameon=False, fontsize=8, loc=\"upper left\")\nax2.plot(ang, J, \"k-\", lw=1.0, label=\"Fisher criterion of the direction\")\nax2.axvline(w_ang, color=\"black\", ls=\"--\", lw=0.9,\n            label=f\"LDA direction ({w_ang:.0f} degrees)\")\nax2.axvline(np.rad2deg(np.arctan2(u1[1], u1[0])) % 180.0, color=\"black\",\n            ls=\":\", lw=0.9, label=\"first principal direction\")\nax2.set_xlabel(\"angle of the direction in degrees\")\nax2.set_ylabel(\"ratio of between-class to within-class scatter\")\nax2.set_title(\"the criterion LDA maximizes\")\nax2.legend(frameon=False, fontsize=8)\nfig.tight_layout()\nfig.savefig(OUT_DIR / \"lda.png\", dpi=110)\n\nn_ok = sum(ok for _, ok in report)\nprint(f\"\\n{n_ok}/{len(report)} checks pass\")\nif n_ok != len(report):\n    raise SystemExit(1)"
  }
 ]
}