{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3",
   "language": "python"
  },
  "language_info": {
   "name": "python"
  },
  "colab": {
   "name": "featurespace.ipynb"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# feature space \u2014 Python demo\n\nNumerical companion to the entry [feature space](https://dictionaryofml.org/terms/featurespace.html) of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org/): it recomputes what the entry states and prints one line per check.\n\nNumerical companion to the glossary entry 'featurespace'. The 366 days of 2024 at Krems are the data points. Each day has a label, its maximum daytime temperature. The feature transformation decides which numbers describe a day, and its range is the feature space: the real line when a day is described by its morning minimum temperature alone, the plane when the maximum temperature of the previous day is added, and the finite set {1, ..., 12} when a day is described by its month.\n\nRequires NumPy and Matplotlib only, and uses fixed seeds, so the printed numbers reproduce exactly. Generated from [`pythondemos/featurespace.py`](https://dictionaryofml.org/terms/featurespace.py); CC BY 4.0.\n\nBefore running a downloaded copy, scan it: `uvx bandit -q -t B102,B307,B602,B605 featurespace.py` (B102,B307,B602,B605 are the exec, eval and shell checks of [bandit](https://bandit.readthedocs.io/)) reports nothing: the script runs no shell command and evaluates no generated code."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "# Notebook shim: the script resolves output paths relative to __file__,\n# which a notebook kernel does not define; everything lands in the\n# working directory instead.\nimport os\n__file__ = os.path.join(os.getcwd(), \"featurespace.py\")\nos.makedirs(\"pythondemos\", exist_ok=True)"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "\"\"\"The feature space of the days of one year at Krems an der Donau: the\nsame days in three feature spaces, and the implicit feature space of a\nkernel.\n\nPurpose\n-------\nNumerical companion to the glossary entry 'featurespace'.  The 366 days\nof 2024 at Krems are the data points.  Each day has a label, its maximum\ndaytime temperature.  The feature transformation decides which numbers\ndescribe a day, and its range is the feature space: the real line when a\nday is described by its morning minimum temperature alone, the plane\nwhen the maximum temperature of the previous day is added, and the\nfinite set {1, ..., 12} when a day is described by its month.\n\nThe demo checks the entry's claims: the feature space is the set in\nwhich the feature vectors live, and its geometry is what a method uses;\nthe same days have different nearest neighbors in different feature\nspaces, and a linear hypothesis learned by least squares has a different\ntraining error in each; on a finite feature space a hypothesis is a\ntable of values, every map on it is linear in the one-hot encoding, and\nevery other day of the same month is a nearest neighbor; the polynomial\nkernel of degree two on the plane is the inner product in an explicit\nsix-dimensional feature space, while the Gaussian kernel has no\nfeature space of dimension below the number of data points.\n\nDeterministic: no random numbers are used.  Self-contained: numpy +\nmatplotlib only; the GeoSphere download falls back to the committed CSV.\n\nBlocks\n------\n[B-data]    Download the daily minimum and maximum temperature at Krems\n            (GeoSphere station 3805) for 31 Dec 2023 to 31 Dec 2024 or\n            read the committed copy; 366 days of 2024, each with the\n            maximum temperature of the previous day.\n[B-spaces]  Two feature transformations of the same days: the morning\n            minimum alone (feature space R) and the pair morning\n            minimum, previous day's maximum (feature space R^2).\n[B-nn]      The nearest neighbor of 22 May 2024 in R and in R^2 are\n            different days with very different labels; 346 of the 366\n            days change their nearest neighbor between the two spaces.\n[B-linear]  A linear hypothesis learned by least squares in each space:\n            training MSE 19.2 in R and 9.2 in R^2; the hypotheses on R\n            are the hypotheses on R^2 that ignore the second feature.\n[B-finite]  The finite feature space of the months: the hypothesis that\n            minimizes the training MSE is the table of monthly means,\n            MSE 17.0; least squares on the one-hot encoding learns the\n            same table; under the discrete metric every other May day is\n            a nearest neighbor of 22 May.\n[B-kernel]  The polynomial kernel (1 + x^T x')^2 on R^2 equals the inner\n            product of an explicit map into R^6, so its 366 x 366 matrix\n            of kernel values has rank 6; for the Gaussian kernel with\n            bandwidth 1 the matrix has rank 366.\n[B-plot]    Every 18th day for the figure (21 of 366 days) plus 22 May\n            and its two nearest neighbors, and the least-squares line\n            on R.\n\nOutputs\n-------\nfeaturespace_weather.csv     : date, tmin, tmax, prevtmax (committed copy).\nfeaturespace_points_plot.csv : the drawn days with their role.\nfeaturespace_line_plot.csv   : two points of the least-squares line on R.\nfeaturespace.png             : matplotlib preview (checking only).\n\"\"\"\n\n# /// script\n# requires-python = \">=3.10\"\n# dependencies = [\"matplotlib==3.10.7\", \"numpy==2.3.4\"]\n# ///\n\nimport json\nimport urllib.request\nfrom pathlib import Path\n\nimport numpy as np\nimport matplotlib\n\nmatplotlib.use(\"Agg\")\nimport matplotlib.pyplot as plt\n\nOUT_DIR = Path(__file__).parent\nQUERY = \"2024-05-22\"\n\nreport = []\n\n\ndef check(name, ok):\n    report.append((name, bool(ok)))\n    print(f\"  [{'ok' if ok else 'FAIL'}] {name}\")"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-data]** Download the daily minimum and maximum temperature at Krems (GeoSphere station 3805) for 31 Dec 2023 to 31 Dec 2024 or read the committed copy; 366 days of 2024, each with the maximum temperature of the previous day."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-data] daily minimum and maximum temperature at Krems, 2024\")\nURL = (\"https://dataset.api.hub.geosphere.at/v1/station/historical/\"\n       \"klima-v2-1d?parameters=tlmin,tlmax&station_ids=3805\"\n       \"&start=2023-12-31&end=2024-12-31\")\nCSV = OUT_DIR / \"featurespace_weather.csv\"\ntry:\n    with urllib.request.urlopen(URL, timeout=120) as resp:\n        payload = json.load(resp)\n    params = payload[\"features\"][0][\"properties\"][\"parameters\"]\n    raw = [(stamp[:10], lo, hi)\n           for stamp, lo, hi in zip(payload[\"timestamps\"],\n                                    params[\"tlmin\"][\"data\"],\n                                    params[\"tlmax\"][\"data\"])]\n    if any(lo is None or hi is None for _, lo, hi in raw):\n        raise OSError(\"a temperature is missing\")\n    # every day of 2024 with the maximum temperature of the previous day\n    records = [(d, lo, hi, raw[i - 1][2])\n               for i, (d, lo, hi) in enumerate(raw) if i > 0]\n    with open(CSV, \"w\") as fh:\n        fh.write(\"date,tmin,tmax,prevtmax\\n\")\n        for d, lo, hi, pv in records:\n            fh.write(f\"{d},{lo},{hi},{pv}\\n\")\n    print(f\"    downloaded {len(raw)} days\")\nexcept OSError as err:\n    print(f\"    download failed ({err}); reading the committed CSV\")\n    rows = CSV.read_text().strip().splitlines()[1:]\n    records = [(d, float(a), float(b), float(c))\n               for d, a, b, c in (r.split(\",\") for r in rows)]\n\ndates = [d for d, _, _, _ in records]\ncheck(\"[B-data] 366 days of 2024, each with the previous day's maximum\",\n      len(records) == 366 and dates[0] == \"2024-01-01\"\n      and dates[-1] == \"2024-12-31\")\ntmin = np.array([lo for _, lo, _, _ in records])\ntmax = np.array([hi for _, _, hi, _ in records])\nprevtmax = np.array([pv for _, _, _, pv in records])\nmonths = np.array([int(d[5:7]) for d in dates])\nm = len(dates)\ny = tmax                                   # the label of a day"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-spaces]** Two feature transformations of the same days: the morning minimum alone (feature space R) and the pair morning minimum, previous day's maximum (feature space R^2)."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-spaces] the same days in the feature spaces R and R^2\")\nX_A = tmin[:, None]                        # feature space R\nX_B = np.column_stack([tmin, prevtmax])    # feature space R^2\ncheck(\"[B-spaces] in R every day is a vector with d = 1 entry, in R^2 \"\n      \"with d = 2 entries\",\n      X_A.shape == (m, 1) and X_B.shape == (m, 2))\ncheck(\"[B-spaces] the first entry of a day's vector in R^2 is its vector \"\n      \"in R: the two transformations agree on the shared feature\",\n      np.array_equal(X_B[:, :1], X_A))\ncheck(f\"[B-spaces] the {m} days occupy {len(np.unique(X_A))} points of R \"\n      f\"and {len(np.unique(X_B, axis=0))} points of R^2\",\n      len(np.unique(X_A)) < m and len(np.unique(X_B, axis=0)) == m)\n\n\ndef nearest(X, q):\n    \"\"\"Index of the nearest other point and its Euclidean distance,\n    plus the distance to the second-nearest (to confirm uniqueness).\"\"\"\n    d = np.sqrt(np.sum((X - X[q]) ** 2, axis=1))\n    d[q] = np.inf\n    order = np.argsort(d)\n    return int(order[0]), float(d[order[0]]), float(d[order[1]])"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-nn]** The nearest neighbor of 22 May 2024 in R and in R^2 are different days with very different labels; 346 of the 366 days change their nearest neighbor between the two spaces."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(f\"[B-nn] the nearest neighbor of {QUERY} in R and in R^2\")\nq = dates.index(QUERY)\nnn_A, d_A, d_A2 = nearest(X_A, q)\nnn_B, d_B, d_B2 = nearest(X_B, q)\ncheck(f\"[B-nn] {QUERY}: morning minimum {tmin[q]:.1f}, previous day's \"\n      f\"maximum {prevtmax[q]:.1f}, label (maximum) {y[q]:.1f}\",\n      True)\ncheck(f\"[B-nn] nearest neighbor in R: {dates[nn_A]} (minimum \"\n      f\"{tmin[nn_A]:.1f}, previous maximum {prevtmax[nn_A]:.1f}, label \"\n      f\"{y[nn_A]:.1f}) at distance {d_A:.1f}, unique (next {d_A2:.1f})\",\n      d_A < d_A2)\ncheck(f\"[B-nn] nearest neighbor in R^2: {dates[nn_B]} (minimum \"\n      f\"{tmin[nn_B]:.1f}, previous maximum {prevtmax[nn_B]:.1f}, label \"\n      f\"{y[nn_B]:.1f}) at distance {d_B:.2f}, unique (next {d_B2:.2f})\",\n      d_B < d_B2)\ncheck(\"[B-nn] the two nearest neighbors are different days\",\n      nn_A != nn_B)\ncheck(f\"[B-nn] the label of the neighbor in R misses the label of \"\n      f\"{QUERY} by {abs(y[nn_A] - y[q]):.1f} degrees, the neighbor in R^2 \"\n      f\"by {abs(y[nn_B] - y[q]):.1f}\",\n      abs(y[nn_A] - y[q]) > 10 > abs(y[nn_B] - y[q]))\nchanged = sum(nearest(X_A, i)[0] != nearest(X_B, i)[0] for i in range(m))\ncheck(f\"[B-nn] {changed} of the {m} days have a different nearest \"\n      f\"neighbor in R than in R^2\", changed > 0.9 * m)\n\n\ndef least_squares(X):\n    \"\"\"Linear hypothesis with intercept learned by least squares, and\n    its training MSE.\"\"\"\n    Z = np.column_stack([np.ones(len(X)), X])\n    w = np.linalg.lstsq(Z, y, rcond=None)[0]\n    return w, float(np.mean((y - Z @ w) ** 2))"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-linear]** A linear hypothesis learned by least squares in each space: training MSE 19.2 in R and 9.2 in R^2; the hypotheses on R are the hypotheses on R^2 that ignore the second feature."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-linear] least squares in R and in R^2\")\nw_A, mse_A = least_squares(X_A)\nw_B, mse_B = least_squares(X_B)\ncheck(f\"[B-linear] on R: h(x) = {w_A[0]:.1f} + {w_A[1]:.2f} x, training \"\n      f\"MSE {mse_A:.1f}\", True)\ncheck(f\"[B-linear] on R^2: h(x) = {w_B[0]:.1f} + {w_B[1]:.2f} x_1 + \"\n      f\"{w_B[2]:.2f} x_2, training MSE {mse_B:.1f}\", True)\ncheck(\"[B-linear] the hypothesis on R, read as a map on R^2 that ignores \"\n      \"x_2, is a linear hypothesis on R^2 with the same training MSE\",\n      abs(least_squares(X_B)[1] - mse_B) < 1e-12\n      and abs(np.mean((y - (w_A[0] + w_A[1] * X_B[:, 0])) ** 2) - mse_A)\n      < 1e-12)\ncheck(\"[B-linear] so the smaller feature space never gives a smaller \"\n      \"training MSE: 19.2 > 9.2\", mse_A > mse_B)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-finite]** The finite feature space of the months: the hypothesis that minimizes the training MSE is the table of monthly means, MSE 17.0; least squares on the one-hot encoding learns the same table; under the discrete metric every other May day is a nearest neighbor of 22 May."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-finite] the month of a day as its feature: X = {1, ..., 12}\")\ntable = np.array([y[months == k].mean() for k in range(1, 13)])\nmse_M = float(np.mean((y - table[months - 1]) ** 2))\ncheck(\"[B-finite] the hypothesis minimizing the training MSE is the table \"\n      f\"of the 12 monthly means; its training MSE is {mse_M:.1f}\",\n      all(abs(np.mean((y - (table[months - 1] + delta * (months == k))) ** 2))\n              >= mse_M - 1e-12\n              for k in range(1, 13) for delta in (-0.5, 0.5)))\nonehot = np.eye(12)[months - 1]\nw_M = np.linalg.lstsq(onehot, y, rcond=None)[0]\ncheck(\"[B-finite] least squares on the one-hot encoding learns the same \"\n      \"table: every map on a finite feature space is linear in the \"\n      \"one-hot vectors\",\n      np.allclose(w_M, table))\nsame_month = int(np.sum(months == months[q])) - 1\ncheck(f\"[B-finite] under the discrete metric all {same_month} other May \"\n      f\"days are nearest neighbors of {QUERY}, at distance 0\",\n      same_month == 30)\ncheck(f\"[B-finite] training MSEs: {mse_A:.1f} (R), {mse_M:.1f} (months), \"\n      f\"{mse_B:.1f} (R^2)\", mse_B < mse_M < mse_A)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-kernel]** The polynomial kernel (1 + x^T x')^2 on R^2 equals the inner product of an explicit map into R^6, so its 366 x 366 matrix of kernel values has rank 6; for the Gaussian kernel with bandwidth 1 the matrix has rank 366."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-kernel] polynomial and Gaussian kernel on the feature space R^2\")\nK_poly = (1.0 + X_B @ X_B.T) ** 2\nphi = np.column_stack([np.ones(m), np.sqrt(2) * X_B[:, 0],\n                       np.sqrt(2) * X_B[:, 1], X_B[:, 0] ** 2,\n                       np.sqrt(2) * X_B[:, 0] * X_B[:, 1], X_B[:, 1] ** 2])\ncheck(\"[B-kernel] (1 + x^T x')^2 equals the inner product of the explicit \"\n      \"map into R^6 for every pair of days\",\n      np.allclose(K_poly, phi @ phi.T, rtol=1e-9, atol=1e-6))\nrank_poly = np.linalg.matrix_rank(K_poly)\ncheck(f\"[B-kernel] the {m} x {m} matrix of polynomial kernel values has \"\n      f\"rank {rank_poly} = dimension of that feature space\", rank_poly == 6)\nsq_dist = np.sum((X_B[:, None, :] - X_B[None, :, :]) ** 2, axis=-1)\nK_gauss = np.exp(-sq_dist / 2.0)           # bandwidth 1 degree\nrank_gauss = np.linalg.matrix_rank(K_gauss)\ncheck(f\"[B-kernel] the matrix of Gaussian kernel values (bandwidth 1) has \"\n      f\"rank {rank_gauss}: no feature space of dimension below {m} \"\n      f\"realizes it\", rank_gauss == m)\ncheck(\"[B-kernel] both matrices are positive semi-definite\",\n      np.linalg.eigvalsh(K_poly).min() > -1e-6\n      and np.linalg.eigvalsh(K_gauss).min() > -1e-9)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-plot]** Every 18th day for the figure (21 of 366 days) plus 22 May and its two nearest neighbors, and the least-squares line on R."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-plot] thinned scatter, the query day and its neighbors, the line\")\nidx = np.arange(0, m, 18)\nroles = {q: 1, nn_A: 2, nn_B: 3}\ndrawn = sorted(set(idx) | set(roles))\nwith open(OUT_DIR / \"featurespace_points_plot.csv\", \"w\") as fh:\n    fh.write(\"date,tmin,prevtmax,tmax,role\\n\")\n    for i in drawn:\n        fh.write(f\"{dates[i]},{tmin[i]:.1f},{prevtmax[i]:.1f},{tmax[i]:.1f},\"\n                 f\"{roles.get(i, 0)}\\n\")\ncheck(f\"[B-plot] every 18th day ({len(idx)} of the {m} days) plus \"\n      f\"{QUERY} and its two nearest neighbors: {len(drawn)} days drawn\",\n      len(idx) == 21 and all(i in drawn for i in roles)\n      and len(drawn) == len(set(idx) | set(roles)))\nxs = np.array([tmin.min() - 1, tmin.max() + 1])\nwith open(OUT_DIR / \"featurespace_line_plot.csv\", \"w\") as fh:\n    fh.write(\"tmin,yhat\\n\")\n    for x in xs:\n        fh.write(f\"{x:.1f},{w_A[0] + w_A[1] * x:.3f}\\n\")\n\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(9.4, 3.9))\nothers = [i for i in drawn if roles.get(i, 0) == 0]\nmarks = ((others, \"o\", \"none\", \"every 18th day\"),\n         ([q], \"s\", \"black\", f\"{QUERY}\"),\n         ([nn_A], \"^\", \"none\", f\"nearest in R: {dates[nn_A]}\"),\n         ([nn_B], \"D\", \"none\", f\"nearest in R^2: {dates[nn_B]}\"))\nfor sel, mk, face, lab in marks:\n    ax1.scatter(tmin[sel], tmax[sel], marker=mk, facecolors=face,\n                edgecolors=\"black\", s=34, label=lab)\n    ax2.scatter(tmin[sel], prevtmax[sel], marker=mk, facecolors=face,\n                edgecolors=\"black\", s=34, label=lab)\nax1.plot(xs, w_A[0] + w_A[1] * xs, \"k-\", lw=1,\n         label=\"least squares on R\")\nax1.set_xlabel(\"morning minimum temperature (\u00b0C) = the feature space R\")\nax1.set_ylabel(\"label: maximum daytime temperature (\u00b0C)\")\nax1.set_title(\"feature space R: the horizontal axis\")\nax1.legend(frameon=False, fontsize=7)\nax2.plot([tmin[q], tmin[nn_B]], [prevtmax[q], prevtmax[nn_B]], \"k--\", lw=0.8)\nax2.set_xlabel(\"morning minimum temperature (\u00b0C)\")\nax2.set_ylabel(\"previous day's maximum temperature (\u00b0C)\")\nax2.set_title(\"feature space R^2: the plane\")\nax2.legend(frameon=False, fontsize=7)\nfig.tight_layout()\nfig.savefig(OUT_DIR / \"featurespace.png\", dpi=110)\n\nn_ok = sum(ok for _, ok in report)\nprint(f\"\\n{n_ok}/{len(report)} checks pass\")\nif n_ok != len(report):\n    raise SystemExit(1)"
  }
 ]
}