{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3",
   "language": "python"
  },
  "language_info": {
   "name": "python"
  },
  "colab": {
   "name": "dob.ipynb"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# degree of belonging \u2014 Python demo\n\nNumerical companion to the entry [degree of belonging](https://dictionaryofml.org/terms/dob.html) of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org/): it recomputes what the entry states and prints one line per check.\n\nNumerical companion to the glossary entry 'dob'. The 366 days of 2024 at Krems are data points with the feature vector (morning minimum temperature, maximum daytime temperature). A soft clustering assigns each day a degree of belonging to every cluster; a hard clustering is the special case where each day belongs to exactly one cluster.\n\nRequires NumPy and Matplotlib only, and uses fixed seeds, so the printed numbers reproduce exactly. Generated from [`pythondemos/dob.py`](https://dictionaryofml.org/terms/dob.py); CC BY 4.0.\n\nBefore running a downloaded copy, scan it: `uvx bandit -q -t B102,B307,B602,B605 dob.py` (B102,B307,B602,B605 are the exec, eval and shell checks of [bandit](https://bandit.readthedocs.io/)) reports nothing: the script runs no shell command and evaluates no generated code."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "# Notebook shim: the script resolves output paths relative to __file__,\n# which a notebook kernel does not define; everything lands in the\n# working directory instead.\nimport os\n__file__ = os.path.join(os.getcwd(), \"dob.py\")\nos.makedirs(\"pythondemos\", exist_ok=True)"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "\"\"\"The degree of belonging of each day of one year at Krems an der Donau\nto a cold cluster and to a warm cluster.\n\nPurpose\n-------\nNumerical companion to the glossary entry 'dob'.  The 366 days of 2024 at\nKrems are data points with the feature vector (morning minimum\ntemperature, maximum daytime temperature).  A soft clustering assigns\neach day a degree of belonging to every cluster; a hard clustering is the\nspecial case where each day belongs to exactly one cluster.\n\nThe demo checks the entry's claims: the two degrees of belonging of a\nday are nonnegative and sum to one, so the vector of degrees of\nbelonging lies in the probability simplex; the posterior probability of\nthe cluster index under a two-component Gaussian mixture model (GMM) is\nsuch a vector; the hard clustering delivered by k-means is the special\ncase where the vector is a vertex of the simplex, with one entry equal to\none and the other equal to zero; the mean of a GMM component is the\naverage of the feature vectors weighted by the degrees of belonging, and\nthe cluster probability is their average, so that with hard assignments\nthe weighted average is the mean of the assigned days; and under a common\ncovariance matrix sigma^2 I with equal cluster probabilities the degrees\nof belonging turn into zero or one as sigma^2 shrinks.\n\nDeterministic: EM starts from the k-means solution, which Lloyd's\nalgorithm reaches from the means of the colder and the warmer half of the\ndays.  Self-contained: numpy + matplotlib only; the GeoSphere download\nfalls back to the committed CSV.\n\nBlocks\n------\n[B-data]      Download the daily minimum and maximum temperature of 2024\n              at Krems (GeoSphere station 3805) or read the committed\n              copy; 366 days, no missing value.\n[B-hard]      k-means with k = 2 by Lloyd's algorithm: the degrees of\n              belonging of a day are 0 or 1 and sum to one, a vertex of\n              the probability simplex; the cold and the warm cluster.\n[B-gmm]       EM for a two-component GMM: the degrees of belonging are the\n              posterior probabilities of the cluster index, lie in [0, 1]\n              and sum to one; the number of days whose larger degree of\n              belonging is below 0.8, by month; the most ambiguous day.\n[B-centroid]  The GMM mean is the average of the days weighted by their\n              degrees of belonging and the cluster probability is their\n              average; with the 0/1 degrees of k-means the weighted\n              average is the mean of the assigned days.\n[B-limit]     Equal cluster probabilities, covariance sigma^2 I and the\n              k-means centroids as means: the share of days with a degree\n              of belonging between 0.2 and 0.8 for several sigma^2, and\n              the agreement with the k-means assignment as sigma^2 -> 0.\n[B-plot]      Every 18th day for the figure (21 of 366 days) with its\n              degrees of belonging, the two GMM means and one-standard-\n              deviation ellipses, and the degree of belonging to the warm\n              cluster of every day of the year.\n\nOutputs\n-------\ndob_weather.csv      : date, tmin, tmax (committed copy).\ndob_points_plot.csv  : the 21 drawn days with their degrees of belonging.\ndob_means.csv        : the two GMM means and the two k-means centroids.\ndob_ellipse1.csv, dob_ellipse2.csv : one-standard-deviation contours.\ndob_year.csv         : day of the year, degree of belonging to the warm\n                       cluster under the GMM, and the k-means assignment.\ndob.png              : matplotlib preview (checking only).\n\"\"\"\n\n# /// script\n# requires-python = \">=3.10\"\n# dependencies = [\"matplotlib==3.10.7\", \"numpy==2.3.4\"]\n# ///\n\nimport json\nimport urllib.request\nfrom pathlib import Path\n\nimport numpy as np\nimport matplotlib\n\nmatplotlib.use(\"Agg\")\nimport matplotlib.pyplot as plt\n\nOUT_DIR = Path(__file__).parent\n\nreport = []\n\n\ndef check(name, ok):\n    report.append((name, bool(ok)))\n    print(f\"  [{'ok' if ok else 'FAIL'}] {name}\")"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-data]** Download the daily minimum and maximum temperature of 2024 at Krems (GeoSphere station 3805) or read the committed copy; 366 days, no missing value."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-data] daily minimum and maximum temperature at Krems, 2024\")\nURL = (\"https://dataset.api.hub.geosphere.at/v1/station/historical/\"\n       \"klima-v2-1d?parameters=tlmin,tlmax&station_ids=3805\"\n       \"&start=2024-01-01&end=2024-12-31\")\nCSV = OUT_DIR / \"dob_weather.csv\"\ntry:\n    with urllib.request.urlopen(URL, timeout=120) as resp:\n        payload = json.load(resp)\n    params = payload[\"features\"][0][\"properties\"][\"parameters\"]\n    records = [(stamp[:10], lo, hi)\n               for stamp, lo, hi in zip(payload[\"timestamps\"],\n                                        params[\"tlmin\"][\"data\"],\n                                        params[\"tlmax\"][\"data\"])]\n    with open(CSV, \"w\") as fh:\n        fh.write(\"date,tmin,tmax\\n\")\n        for day, lo, hi in records:\n            fh.write(f\"{day},{lo},{hi}\\n\")\n    print(f\"    downloaded {len(records)} days\")\nexcept OSError as err:\n    print(f\"    download failed ({err}); reading the committed CSV\")\n    rows = CSV.read_text().strip().splitlines()[1:]\n    records = [(d, float(a), float(b))\n               for d, a, b in (r.split(\",\") for r in rows)]\n\ncheck(\"[B-data] 366 days for 2024\", len(records) == 366)\ncheck(\"[B-data] no day is missing a temperature\",\n      all(lo is not None and hi is not None for _, lo, hi in records))\nX = np.array([[lo, hi] for _, lo, hi in records], dtype=float)\nm, d = X.shape\nk = 2\nmonths = np.array([int(day[5:7]) for day, _, _ in records])\nMONTH = [\"Jan\", \"Feb\", \"Mar\", \"Apr\", \"May\", \"Jun\", \"Jul\", \"Aug\", \"Sep\",\n         \"Oct\", \"Nov\", \"Dec\"]\n\n\ndef sq_dist(X, W):\n    \"\"\"Squared Euclidean distance of every row of X to every row of W.\"\"\"\n    return ((X[:, None, :] - W[None, :, :]) ** 2).sum(axis=2)\n\n\ndef one_hot(labels):\n    \"\"\"The degree-of-belonging vectors of a hard clustering: one entry 1.\"\"\"\n    U = np.zeros((len(labels), k))\n    U[np.arange(len(labels)), labels] = 1.0\n    return U\n\n\ndef in_simplex(U, tol=1e-9):\n    return bool(np.all(U >= -tol) and np.allclose(U.sum(axis=1), 1.0,\n                                                  atol=tol))"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-hard]** k-means with k = 2 by Lloyd's algorithm: the degrees of belonging of a day are 0 or 1 and sum to one, a vertex of the probability simplex; the cold and the warm cluster."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-hard] k-means with k = 2: hard assignments\")\norder = np.argsort(X[:, 0])\nW = np.array([X[order[: m // 2]].mean(axis=0),\n              X[order[m // 2:]].mean(axis=0)])\nlabels = np.argmin(sq_dist(X, W), axis=1)\nfor _ in range(500):\n    W = np.array([X[labels == j].mean(axis=0) for j in range(k)])\n    new = np.argmin(sq_dist(X, W), axis=1)\n    if np.array_equal(new, labels):\n        break\n    labels = new\n# cluster 0 = cold (smaller maximum temperature), cluster 1 = warm\nif W[0, 1] > W[1, 1]:\n    W = W[::-1]\n    labels = 1 - labels\nU_hard = one_hot(labels)\ncheck(\"[B-hard] every degree of belonging of k-means is 0 or 1\",\n      np.all(np.isin(U_hard, [0.0, 1.0])))\ncheck(\"[B-hard] the two degrees of a day sum to one: a vertex of the \"\n      \"probability simplex\", in_simplex(U_hard)\n      and np.all(U_hard.max(axis=1) == 1.0))\nn_cold, n_warm = int((labels == 0).sum()), int((labels == 1).sum())\ncheck(f\"[B-hard] cold centroid ({W[0, 0]:.1f}, {W[0, 1]:.1f}) with \"\n      f\"{n_cold} days, warm centroid ({W[1, 0]:.1f}, {W[1, 1]:.1f}) with \"\n      f\"{n_warm} days\", n_cold + n_warm == m and W[0, 1] < W[1, 1])"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-gmm]** EM for a two-component GMM: the degrees of belonging are the posterior probabilities of the cluster index, lie in [0, 1] and sum to one; the number of days whose larger degree of belonging is below 0.8, by month; the most ambiguous day."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-gmm] EM for a two-component GMM, started from k-means\")\n\n\ndef gauss_pdf(X, mu, S):\n    diff = X - mu\n    Sinv = np.linalg.inv(S)\n    q = np.einsum(\"ij,jk,ik->i\", diff, Sinv, diff)\n    return np.exp(-0.5 * q) / (2 * np.pi * np.sqrt(np.linalg.det(S)))\n\n\ndef posterior(X, p, mus, Ss):\n    \"\"\"The degree of belonging as the posterior probability of the\n    cluster index: Bayes' rule with the mixture density as denominator.\"\"\"\n    joint = np.stack([p[j] * gauss_pdf(X, mus[j], Ss[j]) for j in range(k)],\n                     axis=1)\n    return joint / joint.sum(axis=1, keepdims=True), joint\n\n\ndef neg_loglik(joint):\n    return float(-np.mean(np.log(joint.sum(axis=1))))\n\n\np = np.array([n_cold, n_warm]) / m\nmus = W.copy()\nSs = [np.cov(X[labels == j].T, bias=True) for j in range(k)]\nnll = []\nfor it in range(500):\n    U, joint = posterior(X, p, mus, Ss)                   # E-step\n    nll.append(neg_loglik(joint))\n    Nj = U.sum(axis=0)\n    p = Nj / m                                           # M-step\n    mus = (U.T @ X) / Nj[:, None]\n    Ss = [((U[:, j, None] * (X - mus[j])).T @ (X - mus[j])) / Nj[j]\n          for j in range(k)]\n    if it > 1 and nll[-2] - nll[-1] < 1e-10:\n        break\nU, joint = posterior(X, p, mus, Ss)\nif mus[0, 1] > mus[1, 1]:   # keep cluster 0 = cold\n    p, mus, Ss, U = p[::-1], mus[::-1], Ss[::-1], U[:, ::-1]\ncheck(f\"[B-gmm] the negative log-likelihood never increases over \"\n      f\"{len(nll) - 1} iterations ({nll[0]:.3f} -> {nll[-1]:.3f})\",\n      all(b <= a + 1e-9 for a, b in zip(nll, nll[1:])))\ncheck(\"[B-gmm] every degree of belonging lies in [0, 1]\",\n      np.all((U >= 0) & (U <= 1)))\ncheck(\"[B-gmm] the two degrees of a day sum to one: a vector of the \"\n      \"probability simplex\", in_simplex(U))\npdfs = np.stack([gauss_pdf(X, mus[j], Ss[j]) for j in range(k)], axis=1)\nU_bayes = p * pdfs / (pdfs @ p)[:, None]\ncheck(\"[B-gmm] the degree of belonging equals p_c N(x; mu_c, C_c) divided \"\n      \"by the mixture density (Bayes' rule)\", np.allclose(U, U_bayes))\ncheck(f\"[B-gmm] learned means ({mus[0, 0]:.1f}, {mus[0, 1]:.1f}) and \"\n      f\"({mus[1, 0]:.1f}, {mus[1, 1]:.1f}), cluster probabilities \"\n      f\"{p[0]:.2f} and {p[1]:.2f}\", mus[0, 1] < mus[1, 1])\namb = U.max(axis=1) < 0.8\nby_month = {MONTH[mo - 1]: int(np.sum(amb & (months == mo)))\n            for mo in range(1, 13) if np.sum(amb & (months == mo)) > 0}\ncheck(f\"[B-gmm] {int(amb.sum())} of the {m} days have no degree of \"\n      f\"belonging above 0.8: \" + \", \".join(f\"{v} in {mo}\"\n                                          for mo, v in by_month.items()),\n      0 < amb.sum() < m / 4)\ncheck(\"[B-gmm] no such day lies in November, December, January or \"\n      \"February\", not np.any(amb & np.isin(months, [11, 12, 1, 2])))\ni_amb = int(np.argmin(np.abs(U[:, 1] - 0.5)))\ndate_amb = records[i_amb][0]\ncheck(f\"[B-gmm] the most ambiguous day is {date_amb} with temperatures \"\n      f\"({X[i_amb, 0]:.1f}, {X[i_amb, 1]:.1f}) and degrees of belonging \"\n      f\"({U[i_amb, 0]:.2f}, {U[i_amb, 1]:.2f})\",\n      abs(U[i_amb, 1] - 0.5) < 0.05)\ngmm_hard = np.argmax(U, axis=1)\nagree = float(np.mean(gmm_hard == labels))\ncheck(f\"[B-gmm] the larger degree of belonging names the k-means cluster \"\n      f\"on {100 * agree:.1f}% of the days\", agree > 0.9)\nsure = ~amb\ncheck(f\"[B-gmm] a day with a degree of belonging above 0.8 is in the same \"\n      f\"cluster as under k-means on {100 * np.mean(gmm_hard[sure] == labels[sure]):.1f}% \"\n      f\"of these days\", np.mean(gmm_hard[sure] == labels[sure]) > 0.95)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-centroid]** The GMM mean is the average of the days weighted by their degrees of belonging and the cluster probability is their average; with the 0/1 degrees of k-means the weighted average is the mean of the assigned days."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-centroid] weighted means and cluster probabilities\")\nmu_w = (U.T @ X) / U.sum(axis=0)[:, None]\ncheck(\"[B-centroid] each GMM mean is the average of the days weighted by \"\n      \"their degrees of belonging\", np.allclose(mu_w, mus, atol=1e-6))\ncheck(\"[B-centroid] each cluster probability is the average degree of \"\n      \"belonging over the days\", np.allclose(U.mean(axis=0), p, atol=1e-6))\nmu_hard = (U_hard.T @ X) / U_hard.sum(axis=0)[:, None]\ncheck(\"[B-centroid] with the 0/1 degrees of k-means the same weighted \"\n      \"average is the mean of the assigned days, the k-means centroid\",\n      np.allclose(mu_hard, W)\n      and all(np.allclose(mu_hard[j], X[labels == j].mean(axis=0))\n              for j in range(k)))\ncheck(f\"[B-centroid] the GMM means differ from the k-means centroids by \"\n      f\"at most {np.abs(mus - W).max():.1f} degrees\",\n      np.abs(mus - W).max() < 5)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-limit]** Equal cluster probabilities, covariance sigma^2 I and the k-means centroids as means: the share of days with a degree of belonging between 0.2 and 0.8 for several sigma^2, and the agreement with the k-means assignment as sigma^2 -> 0."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-limit] equal probabilities, covariance sigma^2 I, k-means means\")\nSIGMA2 = [1.0, 50.0, 1000.0]\nshare = {}\nfor s2 in SIGMA2:\n    logit = (sq_dist(X, W)[:, 0] - sq_dist(X, W)[:, 1]) / (2 * s2)\n    u_warm = 1.0 / (1.0 + np.exp(-logit))\n    U_s = np.stack([1 - u_warm, u_warm], axis=1)\n    share[s2] = (float(np.mean((U_s.min(axis=1) > 0.2))),\n                 float(np.mean(np.argmax(U_s, axis=1) == labels)),\n                 float(np.abs(U_s - U_hard).max()))\ncheck(\"[B-limit] share of days with both degrees above 0.2: \"\n      + \", \".join(f\"sigma^2 = {s2:g}: {100 * v[0]:.0f}%\"\n                  for s2, v in share.items()),\n      share[SIGMA2[0]][0] < share[SIGMA2[1]][0] < share[SIGMA2[2]][0])\ncheck(f\"[B-limit] sigma^2 = {SIGMA2[0]:g}: every degree of belonging is \"\n      f\"within {share[SIGMA2[0]][2]:.2f} of the 0/1 value of k-means\",\n      share[SIGMA2[0]][2] < 0.05)\ncheck(\"[B-limit] the larger degree names the k-means cluster for every \"\n      \"sigma^2\", all(v[1] == 1.0 for v in share.values()))\ncheck(f\"[B-limit] sigma^2 = {SIGMA2[2]:g}: no degree of belonging is \"\n      f\"above {np.max(1 / (1 + np.exp(-np.abs((sq_dist(X, W)[:, 0] - sq_dist(X, W)[:, 1]) / (2 * SIGMA2[2]))))):.2f}\",\n      share[SIGMA2[2]][0] > 0.9)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-plot]** Every 18th day for the figure (21 of 366 days) with its degrees of belonging, the two GMM means and one-standard- deviation ellipses, and the degree of belonging to the warm cluster of every day of the year."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-plot] thinned scatter, means, ellipses, degree over the year\")\nidx = np.arange(0, m, 18)\ncls = np.where(U[:, 0] > 0.8, 0, np.where(U[:, 1] > 0.8, 1, 2))\nwith open(OUT_DIR / \"dob_points_plot.csv\", \"w\") as fh:\n    fh.write(\"tmin,tmax,ucold,uwarm,class\\n\")\n    for i in idx:\n        fh.write(f\"{X[i, 0]:.1f},{X[i, 1]:.1f},{U[i, 0]:.3f},{U[i, 1]:.3f},\"\n                 f\"{cls[i]}\\n\")\ncheck(f\"[B-plot] every 18th day: {len(idx)} of the {m} days are drawn, \"\n      f\"{int(np.sum(cls[idx] == 2))} of them with both degrees below 0.8\",\n      len(idx) == 21 and np.sum(cls[idx] == 2) >= 1)\nwith open(OUT_DIR / \"dob_means.csv\", \"w\") as fh:\n    fh.write(\"kindid,kind,cluster,tmin,tmax\\n\")\n    for j in range(k):\n        fh.write(f\"0,gmm,{j},{mus[j, 0]:.4f},{mus[j, 1]:.4f}\\n\")\n    for j in range(k):\n        fh.write(f\"1,kmeans,{j},{W[j, 0]:.4f},{W[j, 1]:.4f}\\n\")\nt = np.linspace(0, 2 * np.pi, 121)\nfor j in range(k):\n    Lc = np.linalg.cholesky(Ss[j])\n    E = mus[j] + (Lc @ np.stack([np.cos(t), np.sin(t)])).T\n    with open(OUT_DIR / f\"dob_ellipse{j + 1}.csv\", \"w\") as fh:\n        fh.write(\"x1,x2\\n\")\n        for a, b in E:\n            fh.write(f\"{a:.4f},{b:.4f}\\n\")\nwith open(OUT_DIR / \"dob_year.csv\", \"w\") as fh:\n    fh.write(\"day,date,uwarm,hard\\n\")\n    for i in range(m):\n        fh.write(f\"{i + 1},{records[i][0]},{U[i, 1]:.4f},{labels[i]}\\n\")\n\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(9.6, 3.9))\nfor c, mk, lab in ((0, \"o\", \"cold days (degree > 0.8)\"),\n                   (1, \"s\", \"warm days (degree > 0.8)\"),\n                   (2, \"^\", \"both degrees below 0.8\")):\n    sel = idx[cls[idx] == c]\n    ax1.scatter(X[sel, 0], X[sel, 1], marker=mk, facecolors=\"none\",\n                edgecolors=\"black\", s=30, label=lab)\nfor j, ls in ((0, \"-\"), (1, \"--\")):\n    Lc = np.linalg.cholesky(Ss[j])\n    E = mus[j] + (Lc @ np.stack([np.cos(t), np.sin(t)])).T\n    ax1.plot(E[:, 0], E[:, 1], color=\"black\", ls=ls, lw=1.0)\nax1.scatter(mus[:, 0], mus[:, 1], marker=\"x\", color=\"black\", s=60,\n            label=\"GMM means\")\nax1.set_xlabel(\"morning minimum temperature (\u00b0C)\")\nax1.set_ylabel(\"maximum daytime temperature (\u00b0C)\")\nax1.set_title(\"every 18th day and its degrees of belonging\")\nax1.legend(frameon=False, fontsize=8)\nax2.plot(np.arange(1, m + 1), U[:, 1], color=\"black\", lw=1.0,\n         label=\"GMM (solid)\")\nax2.step(np.arange(1, m + 1), labels, color=\"black\", ls=\"--\", lw=0.9,\n         where=\"mid\", label=\"k-means (dashed)\")\nax2.set_xlabel(\"day of the year 2024\")\nax2.set_ylabel(\"degree of belonging to the warm cluster\")\nax2.set_title(\"soft against hard assignment over the year\")\nax2.legend(frameon=False, fontsize=8, loc=\"center right\")\nfig.tight_layout()\nfig.savefig(OUT_DIR / \"dob.png\", dpi=110)\n\nn_ok = sum(ok for _, ok in report)\nprint(f\"\\n{n_ok}/{len(report)} checks pass\")\nif n_ok != len(report):\n    raise SystemExit(1)"
  }
 ]
}