{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3",
   "language": "python"
  },
  "language_info": {
   "name": "python"
  },
  "colab": {
   "name": "scatterplot.ipynb"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# scatterplot \u2014 Python demo\n\nNumerical companion to the entry [scatterplot](https://dictionaryofml.org/terms/scatterplot.html) of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org/): it recomputes what the entry states and prints one line per check.\n\nNumerical companion to the glossary entry 'scatterplot'. The 366 days of 2024 at Krems (GeoSphere station 3805) are data points. Each day carries the morning minimum temperature as its feature and the maximum daytime temperature as its label, so a scatterplot of the two puts one marker per day in the plane and a hypothesis that predicts the label from the feature is a curve in the same plane.\n\nRequires NumPy and Matplotlib only, and uses fixed seeds, so the printed numbers reproduce exactly. Generated from [`pythondemos/scatterplot.py`](https://dictionaryofml.org/terms/scatterplot.py); CC BY 4.0.\n\nBefore running a downloaded copy, scan it: `uvx bandit -q -t B102,B307,B602,B605 scatterplot.py` (B102,B307,B602,B605 are the exec, eval and shell checks of [bandit](https://bandit.readthedocs.io/)) reports nothing: the script runs no shell command and evaluates no generated code."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "# Notebook shim: the script resolves output paths relative to __file__,\n# which a notebook kernel does not define; everything lands in the\n# working directory instead.\nimport os\n__file__ = os.path.join(os.getcwd(), \"scatterplot.py\")\nos.makedirs(\"pythondemos\", exist_ok=True)"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "\"\"\"The scatterplot of a dataset with two numeric features: the days of 2024\nat Krems an der Donau drawn as markers, before and after a feature\ntransformation.\n\nPurpose\n-------\nNumerical companion to the glossary entry 'scatterplot'.  The 366 days of\n2024 at Krems (GeoSphere station 3805) are data points.  Each day carries\nthe morning minimum temperature as its feature and the maximum daytime\ntemperature as its label, so a scatterplot of the two puts one marker per\nday in the plane and a hypothesis that predicts the label from the feature\nis a curve in the same plane.\n\nThe demo turns what a reader sees in such a picture into numbers that can\nbe checked: the elongated cloud is a sample correlation coefficient close\nto one and a leading principal direction that carries almost all of the\nsample variance; the day that sits far below the learned line is the\nlargest absolute residual; and the same days drawn in the transformed\nfeatures (daily mean temperature, daily temperature range) form a cloud\nwhose spread is almost entirely along the first axis, which is what a\ndimensionality reduction of these two features exploits.\n\nIt also measures the house rule on how many markers a scatterplot may\ncarry.  Of the 366 days, many lie within half a degree of another day and\ntheir markers overlap at any readable marker size.  Keeping every 20th\ntraining day and every 15th validation and test day leaves 21 days, a\npicture in which far fewer markers overlap, while the sample correlation\ncoefficient of the drawn days stays close to the 366-day value, so the\nthinned picture still shows what the full record shows.\n\nSelf-contained (numpy and matplotlib only), fixed seed for the split; the\nGeoSphere download falls back to the committed CSV.\n\nBlocks\n------\n[B-data]      Daily minimum and maximum temperature at Krems for 2024\n              (366 days, no missing value).\n[B-split]     A fixed-seed split of the 366 days into a training set, a\n              validation set, and a test set.\n[B-learn]     Least squares on the training set gives the linear\n              hypothesis drawn as the solid line of panel (a).\n[B-structure] The cloud of panel (a) as numbers: the sample correlation\n              coefficient of the two features, the share of the sample\n              variance along the leading principal direction, and the day\n              with the largest absolute residual.\n[B-transform] The same days in the transformed features (daily mean, daily\n              range): the sample correlation coefficient falls and the\n              spread concentrates on the first axis.\n[B-thin]      The deterministic thinning rule for the figure, 21 of the\n              366 days: how many markers overlap before and after the\n              thinning, and that the sample correlation coefficient\n              survives it.\n[B-plot]      CSVs for the two panels and the matplotlib preview.\n\nOutputs\n-------\nscatterplot_weather.csv    : date, tmin, tmax for 2024 (committed copy).\nscatterplot_days_plot.csv  : the 21 drawn days with their two features,\n                             their transformed features, and their split.\nscatterplot_line.csv       : two points of the learned hypothesis.\nscatterplot.png            : matplotlib preview (checking only).\n\"\"\"\n\n# /// script\n# requires-python = \">=3.10\"\n# dependencies = [\"matplotlib==3.10.7\", \"numpy==2.3.4\"]\n# ///\n\nimport json\nimport urllib.request\nfrom pathlib import Path\n\nimport numpy as np\nimport matplotlib\n\nmatplotlib.use(\"Agg\")\nimport matplotlib.pyplot as plt\n\nOUT_DIR = Path(__file__).parent\n\nreport = []\n\n\ndef check(name, ok):\n    report.append((name, bool(ok)))\n    print(f\"  [{'ok' if ok else 'FAIL'}] {name}\")\n\n\ndef crowded(points, radius):\n    \"\"\"Number of points that lie within `radius` of another point.\"\"\"\n    diff = points[:, None, :] - points[None, :, :]\n    near = np.sqrt((diff ** 2).sum(axis=2)) < radius\n    np.fill_diagonal(near, False)\n    return int(near.any(axis=1).sum())"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-data]** Daily minimum and maximum temperature at Krems for 2024 (366 days, no missing value)."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-data] daily minimum and maximum temperature at Krems, 2024\")\nURL = (\"https://dataset.api.hub.geosphere.at/v1/station/historical/\"\n       \"klima-v2-1d?parameters=tlmin,tlmax&station_ids=3805\"\n       \"&start=2024-01-01&end=2024-12-31\")\nCSV = OUT_DIR / \"scatterplot_weather.csv\"\ntry:\n    with urllib.request.urlopen(URL, timeout=120) as resp:\n        payload = json.load(resp)\n    params = payload[\"features\"][0][\"properties\"][\"parameters\"]\n    records = [(stamp[:10], lo, hi)\n               for stamp, lo, hi in zip(payload[\"timestamps\"],\n                                        params[\"tlmin\"][\"data\"],\n                                        params[\"tlmax\"][\"data\"])]\n    with open(CSV, \"w\") as fh:\n        fh.write(\"date,tmin,tmax\\n\")\n        for day, lo, hi in records:\n            fh.write(f\"{day},{lo},{hi}\\n\")\n    print(f\"    downloaded {len(records)} days\")\nexcept OSError as err:\n    print(f\"    download failed ({err}); reading the committed CSV\")\n    rows = CSV.read_text().strip().splitlines()[1:]\n    records = [(d, float(a), float(b))\n               for d, a, b in (r.split(\",\") for r in rows)]\n\ncheck(\"[B-data] 366 days for 2024\", len(records) == 366)\ncheck(\"[B-data] no day is missing a temperature\",\n      all(lo is not None and hi is not None for _, lo, hi in records))\ndates = [d for d, _, _ in records]\nx = np.array([lo for _, lo, _ in records], dtype=float)\ny = np.array([hi for _, _, hi in records], dtype=float)\nm = len(x)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-split]** A fixed-seed split of the 366 days into a training set, a validation set, and a test set."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-split] fixed-seed split of the 366 days\")\nrng = np.random.default_rng(2024)\nperm = rng.permutation(m)\nm_train, m_val = 220, 73\nsplit = np.empty(m, dtype=int)                 # 0 train, 1 val, 2 test\nsplit[perm[:m_train]] = 0\nsplit[perm[m_train:m_train + m_val]] = 1\nsplit[perm[m_train + m_val:]] = 2\ncounts = [int((split == s).sum()) for s in (0, 1, 2)]\ncheck(f\"[B-split] {counts[0]} training days, {counts[1]} validation days, \"\n      f\"{counts[2]} test days, {sum(counts)} in total\",\n      counts == [220, 73, 73] and sum(counts) == m)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-learn]** Least squares on the training set gives the linear hypothesis drawn as the solid line of panel (a)."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-learn] linear hypothesis learned on the training days\")\ntr = split == 0\nA = np.column_stack([np.ones(int(tr.sum())), x[tr]])\nw0, w1 = np.linalg.lstsq(A, y[tr], rcond=None)[0]\n\n\ndef h(xx):\n    return w0 + w1 * xx\n\n\nres = y - h(x)\ntrainerr = float(np.mean(res[tr] ** 2))\nvalerr = float(np.mean(res[split == 1] ** 2))\ncheck(f\"[B-learn] h(x) = {w0:.2f} + {w1:.3f} x; the average squared \"\n      f\"deviation of a day from this line is {trainerr:.1f} on the training \"\n      f\"days and {valerr:.1f} on the validation days\",\n      0.5 < w1 < 1.5 and trainerr > 0)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-structure]** The cloud of panel (a) as numbers: the sample correlation coefficient of the two features, the share of the sample variance along the leading principal direction, and the day with the largest absolute residual."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-structure] the elongated cloud as numbers\")\nrho = float(np.corrcoef(x, y)[0, 1])\ncheck(f\"[B-structure] sample correlation coefficient of the two features \"\n      f\"{rho:.3f}: the markers of the 366 days lie close to a rising line\",\n      rho > 0.8)\nX = np.column_stack([x, y])\nev = np.sort(np.linalg.eigvalsh(np.cov(X, rowvar=False)))[::-1]\nshare = float(ev[0] / ev.sum())\ncheck(f\"[B-structure] the leading principal direction carries \"\n      f\"{100 * share:.1f} percent of the sample variance of the two \"\n      f\"features\", share > 0.9)\nfar = int(np.argmax(np.abs(res)))\ncheck(f\"[B-structure] the largest absolute residual is {abs(res[far]):.1f} \"\n      f\"degrees, on {dates[far]} (minimum {x[far]:.1f}, maximum \"\n      f\"{y[far]:.1f}, prediction {h(x[far]):.1f})\",\n      abs(res[far]) > 3 * np.sqrt(trainerr) / 2)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-transform]** The same days in the transformed features (daily mean, daily range): the sample correlation coefficient falls and the spread concentrates on the first axis."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-transform] daily mean temperature and daily temperature range\")\nu = (x + y) / 2.0                              # daily mean temperature\nv = y - x                                      # daily temperature range\nrho_t = float(np.corrcoef(u, v)[0, 1])\ncheck(f\"[B-transform] sample correlation coefficient of the transformed \"\n      f\"features {rho_t:.3f}, against {rho:.3f} for the recorded ones\",\n      abs(rho_t) < abs(rho))\nshare_rec = float(np.var(x) / (np.var(x) + np.var(y)))\nshare_u = float(np.var(u) / (np.var(u) + np.var(v)))\ncheck(f\"[B-transform] the daily mean temperature carries \"\n      f\"{100 * share_u:.1f} percent of the sum of the two sample variances, \"\n      f\"the daily range {100 * (1 - share_u):.1f} percent; the recorded \"\n      f\"features split the same sum {100 * share_rec:.1f} to \"\n      f\"{100 * (1 - share_rec):.1f}\", share_u > share_rec)\ncheck(\"[B-transform] the transformation is invertible: the recorded \"\n      \"temperatures are recovered as u - v/2 and u + v/2\",\n      np.allclose(u - v / 2, x) and np.allclose(u + v / 2, y))"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-thin]** The deterministic thinning rule for the figure, 21 of the 366 days: how many markers overlap before and after the thinning, and that the sample correlation coefficient survives it."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-thin] every 20th training day and every 15th validation and \"\n      \"test day\")\nSTRIDE = (20, 15, 15)\nidx = np.sort(np.concatenate(\n    [np.flatnonzero(split == s)[::STRIDE[s]] for s in (0, 1, 2)]))\ndrawn = [int((split[idx] == s).sum()) for s in (0, 1, 2)]\ncheck(f\"[B-thin] every 20th of the {counts[0]} training days, every 15th of \"\n      f\"the {counts[1]} validation days, and every 15th of the {counts[2]} \"\n      f\"test days: {drawn[0]} + {drawn[1]} + {drawn[2]} = {len(idx)} of the \"\n      f\"{m} days are drawn\", drawn == [11, 5, 5] and len(idx) == 21)\nRADIUS = 0.5                                   # degrees\ncrowd_all = crowded(X, RADIUS)\ncrowd_thin = crowded(X[idx], RADIUS)\ncheck(f\"[B-thin] {crowd_all} of the {m} days lie within {RADIUS} degrees of \"\n      f\"another day, so their markers overlap at any readable marker size; \"\n      f\"{crowd_thin} of the {len(idx)} drawn days do\", crowd_thin < crowd_all)\nrho_thin = float(np.corrcoef(x[idx], y[idx])[0, 1])\ncheck(f\"[B-thin] sample correlation coefficient of the drawn days \"\n      f\"{rho_thin:.3f}, within {abs(rho_thin - rho):.3f} of the {m}-day \"\n      f\"value {rho:.3f}\", abs(rho_thin - rho) < 0.1)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-plot]** CSVs for the two panels and the matplotlib preview."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-plot] CSVs for the two panels and the preview\")\nwith open(OUT_DIR / \"scatterplot_days_plot.csv\", \"w\") as fh:\n    fh.write(\"tmin,tmax,tmean,trange,split\\n\")\n    for i in idx:\n        fh.write(f\"{x[i]:.1f},{y[i]:.1f},{u[i]:.2f},{v[i]:.2f},\"\n                 f\"{split[i]}\\n\")\nx_lo, x_hi = float(x.min()), float(x.max())\nwith open(OUT_DIR / \"scatterplot_line.csv\", \"w\") as fh:\n    fh.write(\"tmin,pred\\n\")\n    for xx in (x_lo, x_hi):\n        fh.write(f\"{xx:.2f},{h(xx):.3f}\\n\")\n\nMARKS = ((0, \"o\", \"k\", \"training set\"),\n         (1, \"^\", \"none\", \"validation set\"),\n         (2, \"s\", \"none\", \"test set\"))\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(9.4, 3.9))\nfor s, mk, fc, lab in MARKS:\n    sel = idx[split[idx] == s]\n    ax1.scatter(x[sel], y[sel], marker=mk, facecolors=fc,\n                edgecolors=\"black\", s=28, label=lab)\n    ax2.scatter(u[sel], v[sel], marker=mk, facecolors=fc,\n                edgecolors=\"black\", s=28, label=lab)\nax1.plot([x_lo, x_hi], [h(x_lo), h(x_hi)], \"k-\", lw=0.9,\n         label=\"learned hypothesis\")\nax1.set_xlabel(\"morning minimum temperature (\u00b0C)\")\nax1.set_ylabel(\"maximum daytime temperature (\u00b0C)\")\nax1.set_title(\"(a) the two recorded features\")\nax1.legend(frameon=False, fontsize=8)\nax2.set_xlabel(\"daily mean temperature (\u00b0C)\")\nax2.set_ylabel(\"daily temperature range (\u00b0C)\")\nax2.set_title(\"(b) the two transformed features\")\nax2.legend(frameon=False, fontsize=8)\nfig.tight_layout()\nfig.savefig(OUT_DIR / \"scatterplot.png\", dpi=110)\n\nn_ok = sum(ok for _, ok in report)\nprint(f\"\\n{n_ok}/{len(report)} checks pass\")\nif n_ok != len(report):\n    raise SystemExit(1)"
  }
 ]
}