{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3",
   "language": "python"
  },
  "language_info": {
   "name": "python"
  },
  "colab": {
   "name": "featuretransformation.ipynb"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# feature transformation \u2014 Python demo\n\nNumerical companion to the entry [feature transformation](https://dictionaryofml.org/terms/featuretransformation.html) of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org/): it recomputes what the entry states and prints one line per check.\n\nNumerical companion to the glossary entry 'featuretransformation'. Each data point is one day; its single raw feature is the day of the year, rescaled to the interval [-1, 1], and its label is the maximum daytime temperature. A linear hypothesis of the raw feature is learned by least squares on the 365 days of 2023 (the training set) and judged on the 366 days of 2024 (the validation set). The same method is then applied to the transformed feature vector (x, x^2, ..., x^(d')), so that a linear hypothesis on the new features is a polynomial of the raw one. The demo recomputes the three numbers the entry states: the error of the straight line, the error after the transformation with d' = 5 new features, and the training and validation error over d' = 1, ..., 14, where the training error keeps falling while the validation error turns up.\n\nRequires NumPy and Matplotlib only, and uses fixed seeds, so the printed numbers reproduce exactly. Generated from [`pythondemos/featuretransformation.py`](https://dictionaryofml.org/terms/featuretransformation.py); CC BY 4.0.\n\nBefore running a downloaded copy, scan it: `uvx bandit -q -t B102,B307,B602,B605 featuretransformation.py` (B102,B307,B602,B605 are the exec, eval and shell checks of [bandit](https://bandit.readthedocs.io/)) reports nothing: the script runs no shell command and evaluates no generated code."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "# Notebook shim: the script resolves output paths relative to __file__,\n# which a notebook kernel does not define; everything lands in the\n# working directory instead.\nimport os\n__file__ = os.path.join(os.getcwd(), \"featuretransformation.py\")\nos.makedirs(\"pythondemos\", exist_ok=True)"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "\"\"\"A feature transformation on the daily weather at Krems an der Donau\n(Austria): a linear hypothesis on the raw feature, and the same method on a\npolynomial transformation of it.\n\nPurpose\n-------\nNumerical companion to the glossary entry 'featuretransformation'.  Each\ndata point is one day; its single raw feature is the day of the year,\nrescaled to the interval [-1, 1], and its label is the maximum daytime\ntemperature.  A linear hypothesis of the raw feature is learned by least\nsquares on the 365 days of 2023 (the training set) and judged on the 366\ndays of 2024 (the validation set).  The same method is then applied to the\ntransformed feature vector (x, x^2, ..., x^(d')), so that a linear\nhypothesis on the new features is a polynomial of the raw one.  The demo\nrecomputes the three numbers the entry states: the error of the straight\nline, the error after the transformation with d' = 5 new features, and the\ntraining and validation error over d' = 1, ..., 14, where the training\nerror keeps falling while the validation error turns up.\n\nIt also shows that the number of new features is not the whole choice:\nthe three odd powers x, x^3, x^5 leave the validation error above that of\npredicting the sample mean of the training labels, while the two even\npowers x^2, x^4 alone are better than all five powers together, because\nthe seasonal cycle is nearly symmetric about the middle of the year.\n\nIt verifies the composition claim twice: the two-step evaluation\n(transform the raw feature, then apply the linear hypothesis) agrees with\nthe polynomial evaluated directly, and the composed hypothesis is not an\naffine function of the raw feature, while the one learned on the raw\nfeature is.\n\nSelf-contained (numpy and matplotlib only), fixed seeds; the GeoSphere\ndownload falls back to the committed CSV.\n\nBlocks\n------\n[B-data]  Daily maximum temperature at Krems for 2023 and 2024 (731 days,\n          no missing value), the raw feature and the label.\n[B-raw]   A linear hypothesis of the raw feature, learned by least squares\n          on the 2023 days; its training and validation error against the\n          error of predicting the sample mean of the training labels.\n[B-trafo] The polynomial feature transformation with d' = 5 new features;\n          the training and validation error of a linear hypothesis on the\n          new features, and the two composition checks.\n[B-prune] Training and validation error over d' = 1, ..., 14: the training\n          error falls with d', the validation error is smallest at d' = 5\n          and rises beyond it.\n[B-blind] A transformation fixed in advance can discard what the label\n          depends on: the three odd powers of the raw feature against the\n          two even ones, and the symmetry of the seasonal cycle about the\n          middle of the year that separates them.\n[B-draw]  The thinned set of 20 training days drawn in the figure, the\n          plot CSVs, and the matplotlib preview.\n\nOutputs\n-------\nfeaturetransformation_weather.csv    : date, tmax for 2023 and 2024\n                                       (committed copy of the download).\nfeaturetransformation_days_plot.csv  : every 19th day of 2023, 20 of the\n                                       365, for the plot of panel (a).\nfeaturetransformation_curves.csv     : the two learned hypotheses on a grid\n                                       of days, d' = 1 and d' = 5.\nfeaturetransformation_errors.csv     : d', training error, validation error.\nfeaturetransformation.png            : matplotlib preview (checking only).\n\"\"\"\n\n# /// script\n# requires-python = \">=3.10\"\n# dependencies = [\"matplotlib==3.10.7\", \"numpy==2.3.4\"]\n# ///\n\nimport json\nimport urllib.request\nfrom pathlib import Path\n\nimport numpy as np\nimport matplotlib\n\nmatplotlib.use(\"Agg\")\nimport matplotlib.pyplot as plt\n\nOUT_DIR = Path(__file__).parent\n\nDPRIME_MAX = 14          # largest number of new features examined\nDPRIME_STAR = 5          # the number of new features the entry reports\nTHIN = 19                # every 19th training day is drawn\n\nreport = []\n\n\ndef check(name, ok):\n    report.append((name, bool(ok)))\n    print(f\"  [{'ok' if ok else 'FAIL'}] {name}\")"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-data]** Daily maximum temperature at Krems for 2023 and 2024 (731 days, no missing value), the raw feature and the label."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-data] daily maximum temperature at Krems, 2023 and 2024\")\nURL = (\"https://dataset.api.hub.geosphere.at/v1/station/historical/\"\n       \"klima-v2-1d?parameters=tlmax&station_ids=3805\"\n       \"&start=2023-01-01&end=2024-12-31\")\nCSV = OUT_DIR / \"featuretransformation_weather.csv\"\ntry:\n    with urllib.request.urlopen(URL, timeout=120) as resp:\n        payload = json.load(resp)\n    params = payload[\"features\"][0][\"properties\"][\"parameters\"]\n    records = [(stamp[:10], hi)\n               for stamp, hi in zip(payload[\"timestamps\"],\n                                    params[\"tlmax\"][\"data\"])]\n    with open(CSV, \"w\") as fh:\n        fh.write(\"date,tmax\\n\")\n        for day, hi in records:\n            fh.write(f\"{day},{hi}\\n\")\n    print(f\"    downloaded {len(records)} days\")\nexcept OSError as err:\n    print(f\"    download failed ({err}); reading the committed CSV\")\n    rows = CSV.read_text().strip().splitlines()[1:]\n    records = [(d, float(v)) for d, v in (r.split(\",\") for r in rows)]\n\ncheck(\"[B-data] 731 days for 2023 and 2024\", len(records) == 731)\ncheck(\"[B-data] no day is missing its maximum temperature\",\n      all(hi is not None for _, hi in records))\n\ny_all = np.array([float(hi) for _, hi in records])\nis23 = np.array([d.startswith(\"2023\") for d, _ in records])\ny_tr, y_va = y_all[is23], y_all[~is23]\nm_tr, m_va = len(y_tr), len(y_va)\n# the raw feature of a day: its position in the year, rescaled to [-1, 1]\nday_tr = np.arange(1, m_tr + 1)\nday_va = np.arange(1, m_va + 1)\nx_tr = 2.0 * day_tr / m_tr - 1.0\nx_va = 2.0 * day_va / m_va - 1.0\ncheck(f\"[B-data] the raw feature of a training day runs from \"\n      f\"{x_tr[0]:.4f} to {x_tr[-1]:.4f}; {m_tr} training days and \"\n      f\"{m_va} validation days\",\n      -1.0 < x_tr[0] < -0.99 and abs(x_tr[-1] - 1.0) < 1e-12)\n\n\ndef learn(dprime):\n    \"\"\"Least squares on the transformed training features; returns the\n    weight vector of the linear hypothesis on the d' new features.\"\"\"\n    A = np.vander(x_tr, dprime + 1, increasing=True)\n    return np.linalg.lstsq(A, y_tr, rcond=None)[0]\n\n\ndef errors(w):\n    \"\"\"Training and validation error (average squared error loss).\"\"\"\n    dprime = len(w) - 1\n    tr = np.mean((y_tr - np.vander(x_tr, dprime + 1, increasing=True) @ w) ** 2)\n    va = np.mean((y_va - np.vander(x_va, dprime + 1, increasing=True) @ w) ** 2)\n    return float(tr), float(va)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-raw]** A linear hypothesis of the raw feature, learned by least squares on the 2023 days; its training and validation error against the error of predicting the sample mean of the training labels."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-raw] a linear hypothesis of the raw feature alone\")\nw_raw = learn(1)\ntr_raw, va_raw = errors(w_raw)\nmean_tr = float(np.mean(y_tr))\nva_mean = float(np.mean((y_va - mean_tr) ** 2))\nspan = float(y_tr.max() - y_tr.min())\ncheck(f\"[B-raw] the learned hypothesis is h(x) = {w_raw[0]:.2f} \"\n      f\"{'+' if w_raw[1] >= 0 else '-'} {abs(w_raw[1]):.2f} x: it rises by \"\n      f\"{2 * abs(w_raw[1]):.1f} degrees from the first to the last day of \"\n      f\"the year, against a label range of {span:.1f} degrees\",\n      2 * abs(w_raw[1]) < 0.25 * span)\ncheck(f\"[B-raw] training error {tr_raw:.2f} and validation error \"\n      f\"{va_raw:.2f} squared degrees\", tr_raw > 0 and va_raw > 0)\ncheck(f\"[B-raw] predicting the sample mean {mean_tr:.2f} degrees of the \"\n      f\"training labels gives the validation error {va_mean:.2f}, below the \"\n      f\"{va_raw:.2f} of the straight line, so the raw feature is of no use \"\n      f\"to a linear hypothesis\", va_raw > va_mean)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-trafo]** The polynomial feature transformation with d' = 5 new features; the training and validation error of a linear hypothesis on the new features, and the two composition checks."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(f\"[B-trafo] a polynomial feature transformation with \"\n      f\"{DPRIME_STAR} new features\")\nw_star = learn(DPRIME_STAR)\ntr_star, va_star = errors(w_star)\ncheck(f\"[B-trafo] training error {tr_star:.2f} and validation error \"\n      f\"{va_star:.2f} squared degrees with {DPRIME_STAR} new features\",\n      va_star < va_raw)\ncheck(f\"[B-trafo] the validation error falls from {va_raw:.2f} to \"\n      f\"{va_star:.2f} squared degrees, a factor of \"\n      f\"{va_raw / va_star:.1f}\", va_star < va_raw / 3)\n# composition, step by step: transform the raw feature, then apply the\n# linear hypothesis on the new features\nprobe = np.array([-0.7, -0.2, 0.0, 0.35, 0.8])\ntwo_step = np.array([float(w_star[0]\n                          + np.dot(w_star[1:],\n                                   np.array([xx ** j for j in\n                                             range(1, DPRIME_STAR + 1)])))\n                     for xx in probe])\ndirect = np.polyval(w_star[::-1], probe)\ncheck(f\"[B-trafo] the two-step evaluation (transform, then the linear \"\n      f\"hypothesis) agrees with the polynomial of the raw feature at \"\n      f\"{len(probe)} probe days, largest gap \"\n      f\"{np.max(np.abs(two_step - direct)):.2e}\",\n      np.allclose(two_step, direct, atol=1e-9))\nmid = 0.5 * (probe[0] + probe[-1])\ncomp_mid = float(np.polyval(w_star[::-1], mid))\ncomp_avg = 0.5 * (two_step[0] + two_step[-1])\nline_mid = float(w_raw[0] + w_raw[1] * mid)\nline_avg = 0.5 * ((w_raw[0] + w_raw[1] * probe[0])\n                  + (w_raw[0] + w_raw[1] * probe[-1]))\ncheck(f\"[B-trafo] the composed hypothesis is not affine in the raw \"\n      f\"feature: at the midpoint of two days it gives {comp_mid:.2f} \"\n      f\"degrees against the average {comp_avg:.2f} of its two values\",\n      abs(comp_mid - comp_avg) > 1.0)\ncheck(f\"[B-trafo] the hypothesis learned on the raw feature is affine: \"\n      f\"{line_mid:.4f} against {line_avg:.4f}\",\n      abs(line_mid - line_avg) < 1e-9)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-prune]** Training and validation error over d' = 1, ..., 14: the training error falls with d', the validation error is smallest at d' = 5 and rises beyond it."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-prune] training and validation error over the number of new \"\n      \"features\")\ndprimes = np.arange(1, DPRIME_MAX + 1)\ntr_curve, va_curve = [], []\nfor dp in dprimes:\n    a, b = errors(learn(int(dp)))\n    tr_curve.append(a)\n    va_curve.append(b)\ntr_curve, va_curve = np.array(tr_curve), np.array(va_curve)\nbest = int(dprimes[int(np.argmin(va_curve))])\ncheck(f\"[B-prune] the training error falls from {tr_curve[0]:.2f} to \"\n      f\"{tr_curve[-1]:.2f} squared degrees as d' grows from 1 to \"\n      f\"{DPRIME_MAX}\", np.all(np.diff(tr_curve) < 1e-6))\ncheck(f\"[B-prune] the validation error is smallest at d' = {best} with \"\n      f\"{va_curve.min():.2f} squared degrees\", best == DPRIME_STAR)\ncheck(f\"[B-prune] beyond d' = {best} the validation error rises again, to \"\n      f\"{va_curve[-1]:.2f} at d' = {DPRIME_MAX}, while the training error \"\n      f\"keeps falling\", va_curve[-1] > va_curve.min() + 2.0)\ncheck(f\"[B-prune] the gap between the two errors widens from \"\n      f\"{va_curve[0] - tr_curve[0]:.2f} at d' = 1 to \"\n      f\"{va_curve[-1] - tr_curve[-1]:.2f} at d' = {DPRIME_MAX}\",\n      va_curve[-1] - tr_curve[-1] > va_curve[0] - tr_curve[0])"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-blind]** A transformation fixed in advance can discard what the label depends on: the three odd powers of the raw feature against the two even ones, and the symmetry of the seasonal cycle about the middle of the year that separates them."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-blind] a transformation fixed in advance can discard what the \"\n      \"label depends on\")\n\n\ndef errors_on(cols_tr, cols_va):\n    \"\"\"Training and validation error of least squares on given columns.\"\"\"\n    A = np.column_stack([np.ones(m_tr)] + cols_tr)\n    B = np.column_stack([np.ones(m_va)] + cols_va)\n    w = np.linalg.lstsq(A, y_tr, rcond=None)[0]\n    return (float(np.mean((y_tr - A @ w) ** 2)),\n            float(np.mean((y_va - B @ w) ** 2)))\n\n\ntr_odd, va_odd = errors_on([x_tr, x_tr ** 3, x_tr ** 5],\n                           [x_va, x_va ** 3, x_va ** 5])\ntr_even, va_even = errors_on([x_tr ** 2, x_tr ** 4], [x_va ** 2, x_va ** 4])\n# the label as a function of the raw feature, split into its symmetric and\n# its antisymmetric part about x = 0, the middle of the year; the two parts\n# are uncorrelated, so the average squared deviation of the labels from\n# their sample mean splits between them\ny_mirror = np.interp(-x_tr, x_tr, y_tr)\nsym = 0.5 * (y_tr + y_mirror)\nanti = 0.5 * (y_tr - y_mirror)\nshare = float(sym.var() / (sym.var() + anti.var()))\ncheck(f\"[B-blind] the average squared deviation \"\n      f\"{sym.var() + anti.var():.2f} of the training labels from their \"\n      f\"sample mean splits between the symmetric and the antisymmetric \"\n      f\"part, and matches the {y_tr.var():.2f} computed directly\",\n      abs(sym.var() + anti.var() - y_tr.var()) < 1e-9)\ncheck(f\"[B-blind] the maximum temperature at Krems is nearly symmetric \"\n      f\"about the middle of the year: {100 * share:.0f} percent of that \"\n      f\"squared deviation sits in the symmetric part\", share > 0.7)\ncheck(f\"[B-blind] the three odd powers x, x^3, x^5 are antisymmetric about \"\n      f\"the middle of the year and leave the validation error at \"\n      f\"{va_odd:.2f} squared degrees, above the {va_mean:.2f} of the \"\n      f\"constant hypothesis, although they deliver three new features \"\n      f\"where the raw feature is one\", va_odd > va_mean)\ncheck(f\"[B-blind] the two even powers x^2, x^4 alone give the validation \"\n      f\"error {va_even:.2f} squared degrees, below the {va_star:.2f} of all \"\n      f\"five powers\", va_even < va_star)\ncheck(f\"[B-blind] the training error separates the two choices as well: \"\n      f\"{tr_odd:.2f} squared degrees for the odd powers against \"\n      f\"{tr_even:.2f} for the even ones\", tr_odd > tr_even)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-draw]** The thinned set of 20 training days drawn in the figure, the plot CSVs, and the matplotlib preview."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-draw] the thinned training days, the plot CSVs and the preview\")\nthin = np.arange(0, m_tr, THIN)\ncheck(f\"[B-draw] {len(thin)} of the {m_tr} training days are drawn, \"\n      f\"every {THIN}th in calendar order\", len(thin) <= 21)\nwith open(OUT_DIR / \"featuretransformation_days_plot.csv\", \"w\") as fh:\n    fh.write(\"day,tmax\\n\")\n    for i in thin:\n        fh.write(f\"{day_tr[i]},{y_tr[i]:.1f}\\n\")\ngrid_day = np.linspace(1, m_tr, 200)\ngrid_x = 2.0 * grid_day / m_tr - 1.0\nwith open(OUT_DIR / \"featuretransformation_curves.csv\", \"w\") as fh:\n    fh.write(\"day,raw,poly\\n\")\n    for d, xx in zip(grid_day, grid_x):\n        fh.write(f\"{d:.3f},{np.polyval(w_raw[::-1], xx):.4f},\"\n                 f\"{np.polyval(w_star[::-1], xx):.4f}\\n\")\nwith open(OUT_DIR / \"featuretransformation_errors.csv\", \"w\") as fh:\n    fh.write(\"dprime,train,val\\n\")\n    for dp, a, b in zip(dprimes, tr_curve, va_curve):\n        fh.write(f\"{dp},{a:.4f},{b:.4f}\\n\")\n\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(9.6, 3.9))\nax1.plot(day_tr[thin], y_tr[thin], \"ko\", ms=4,\n         label=\"training day (filled circles)\")\nax1.plot(grid_day, np.polyval(w_raw[::-1], grid_x), \"k-\",\n         label=\"linear hypothesis of the raw feature (solid curve)\")\nax1.plot(grid_day, np.polyval(w_star[::-1], grid_x), \"k--\",\n         label=f\"linear hypothesis of the {DPRIME_STAR} new features \"\n               f\"(dashed curve)\")\nax1.set_xlabel(\"day of the year 2023 (the raw feature)\")\nax1.set_ylabel(\"maximum temperature in degrees Celsius\")\nax1.set_ylim(-3, 48)\nax1.set_title(\"the transformation bends the hypothesis\")\nax1.legend(frameon=False, fontsize=7, loc=\"upper left\")\nax2.plot(dprimes, tr_curve, \"k-o\", ms=4,\n         label=\"training error (solid curve, filled circles)\")\nax2.plot(dprimes, va_curve, \"k--s\", ms=4, mfc=\"white\",\n         label=\"validation error (dashed curve, open squares)\")\nax2.set_xlabel(\"number $d'$ of new features\")\nax2.set_ylabel(\"average squared error loss\")\nax2.set_title(\"more new features, a larger hypothesis space\")\nax2.legend(frameon=False, fontsize=7)\nfig.tight_layout()\nfig.savefig(OUT_DIR / \"featuretransformation.png\", dpi=110)\n\nn_ok = sum(ok for _, ok in report)\nprint(f\"\\n{n_ok}/{len(report)} checks pass\")\nif n_ok != len(report):\n    raise SystemExit(1)"
  }
 ]
}