{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3",
   "language": "python"
  },
  "language_info": {
   "name": "python"
  },
  "colab": {
   "name": "distshift.ipynb"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# distribution shift \u2014 Python demo\n\nNumerical companion to the entry [distribution shift](https://dictionaryofml.org/terms/distshift.html) of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org/): it recomputes what the entry states and prints one line per check.\n\nNumerical companion to the glossary entry 'distshift' (distribution shift). The Finnish Meteorological Institute (FMI) station Helsinki Kaisaniemi (fmisid 100971, 60.18 N, 24.94 E) and the northernmost FMI station, Utsjoki Nuorgam (fmisid 102036, 70.08 N, 27.90 E), record the minimum and maximum air temperature of each day. This script downloads the daily records for June-August 2026 (Kaisaniemi) and December 2025 - February 2026 (Nuorgam) from the FMI open data service and writes them to CSV files, so the exact numbers behind the figure stay on record. Checks pin the downloaded values to the archive, so a change on the server side is caught rather than silently absorbed.\n\nRequires NumPy and Matplotlib only, and uses fixed seeds, so the printed numbers reproduce exactly. Generated from [`pythondemos/distshift.py`](https://dictionaryofml.org/terms/distshift.py); CC BY 4.0."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "# Notebook shim: the script resolves output paths relative to __file__,\n# which a notebook kernel does not define; everything lands in the\n# working directory instead.\nimport os\n__file__ = os.path.join(os.getcwd(), \"distshift.py\")\nos.makedirs(\"pythondemos\", exist_ok=True)"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "\"\"\"Distribution shift between two FMI weather stations: a linear\nhypothesis learned from Helsinki summer days fails on winter days at\nUtsjoki Nuorgam.\n\nPurpose\n-------\nNumerical companion to the glossary entry 'distshift' (distribution\nshift).  The Finnish Meteorological Institute (FMI) station Helsinki\nKaisaniemi (fmisid 100971, 60.18 N, 24.94 E) and the northernmost FMI\nstation, Utsjoki Nuorgam (fmisid 102036, 70.08 N, 27.90 E), record the\nminimum and maximum air temperature of each day.  This script downloads\nthe daily records for June-August 2026 (Kaisaniemi) and December\n2025 - February 2026 (Nuorgam) from the FMI open data service and\nwrites them to CSV files, so the exact numbers behind the figure stay\non record.  Checks pin the downloaded values to the archive, so a\nchange on the server side is caught rather than silently absorbed.\n\nEach data point is one day: its feature is the minimum temperature of\nthat day, its label the maximum temperature of the same day.  A linear\nhypothesis is learned from Helsinki summer days by minimizing the average\nsquared error on the training set; its average squared error on a\nheld-out test set from the same station and season is close to the\ntraining error.  Applied to the winter days at Nuorgam, the same\nhypothesis errs dozens of times worse: every winter feature lies far\nbelow every summer feature (the distribution of the feature values\nhas shifted), and the relation between the two temperatures differs as\nwell (the conditional distribution has shifted too), so the summer line\nextrapolates to above-zero maxima for Arctic winter mornings.\n\nDeterministic: the only randomness is the train/test split of the\nHelsinki days, drawn with a fixed seed.  Self-contained: numpy +\nmatplotlib only (stdlib urllib for the download).\n\nBlocks\n------\n[B-fetch] Download the daily minimum and maximum temperature for\n          Helsinki Kaisaniemi (June-August 2026) and Utsjoki Nuorgam\n          (December 2025 - February 2026) from the FMI open data\n          service; write them to distshift_helsinki.csv and\n          distshift_nuorgam.csv and check them against the archive.\n[B-split] Split the Helsinki days into a training set and a test set\n          with a fixed seed; check the sizes.\n[B-learn] Learn the linear hypothesis by minimizing the average squared\n          error on the training set; check that the fitted coefficients\n          match the archive.\n[B-shift] Average squared error on the training set, the test set, and\n          the Nuorgam winter days; check that the test error stays\n          close to the training error while the Nuorgam error is\n          several times larger, that the two feature ranges do not\n          overlap, that the hypothesis overestimates every winter\n          maximum, and that a line fitted to the winter days has a\n          different slope (the conditional distribution shifted too).\n\nOutputs\n-------\ndistshift_helsinki.csv   : date, tmin, tmax of the Helsinki summer days\ndistshift_nuorgam.csv    : date, tmin, tmax of the Nuorgam winter days\ndistshift_train.csv      : tmin, tmax of the Helsinki training set\ndistshift_test.csv       : tmin, tmax of the Helsinki test set\ndistshift_deploy.csv     : tmin, tmax of the Nuorgam winter days\ndistshift_hypothesis.csv : tmin, tmax along the learned hypothesis\ndistshift_errors.csv     : average squared error on the three sets\ndistshift.png            : preview (checking only) -- the data points,\n                           the learned hypothesis, and the three errors\n\"\"\"\n\nimport re\nimport urllib.request\nfrom pathlib import Path\n\nimport numpy as np\nimport matplotlib\n\nmatplotlib.use(\"Agg\")\nimport matplotlib.pyplot as plt\n\nOUT_DIR = Path(__file__).parent\n\nreport = []                         # collects (check name, pass/fail) pairs\n\n\ndef check(name, ok):                # records and prints one verification\n    report.append((name, bool(ok)))\n    print(f\"  [{'ok' if ok else 'FAIL'}] {name}\")"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-fetch]** Download the daily minimum and maximum temperature for Helsinki Kaisaniemi (June-August 2026) and Utsjoki Nuorgam (December 2025 - February 2026) from the FMI open data service; write them to distshift_helsinki.csv and distshift_nuorgam.csv and check them against the archive."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "def fetch_days(fmisid, start, end):\n    \"\"\"Daily (date, tmin, tmax) records of one station, NaN days dropped.\"\"\"\n    url = (\"https://opendata.fmi.fi/wfs?service=WFS&version=2.0.0\"\n           \"&request=getFeature\"\n           \"&storedquery_id=fmi::observations::weather::daily::simple\"\n           f\"&fmisid={fmisid}&starttime={start}T00:00:00Z\"\n           f\"&endtime={end}T00:00:00Z&parameters=tmin,tmax\")\n    with urllib.request.urlopen(url, timeout=120) as resp:\n        text = resp.read().decode()\n    values = {}                     # date -> {\"tmin\": v, \"tmax\": v}\n    member = (r\"<BsWfs:Time>(\\d{4}-\\d{2}-\\d{2})T[^<]*</BsWfs:Time>\\s*\"\n              r\"<BsWfs:ParameterName>(tmin|tmax)</BsWfs:ParameterName>\\s*\"\n              r\"<BsWfs:ParameterValue>([^<]+)</BsWfs:ParameterValue>\")\n    for day, name, value in re.findall(member, text):\n        values.setdefault(day, {})[name] = float(value)\n    return [(day, rec[\"tmin\"], rec[\"tmax\"])\n            for day, rec in sorted(values.items())\n            if len(rec) == 2 and np.isfinite([rec[\"tmin\"], rec[\"tmax\"]]).all()]\n\n\nhelsinki = fetch_days(100971, \"2026-06-01\", \"2026-08-31\")\nnuorgam = fetch_days(102036, \"2025-12-01\", \"2026-02-28\")\nfor fname, records in ((\"distshift_helsinki.csv\", helsinki),\n                       (\"distshift_nuorgam.csv\", nuorgam)):\n    with open(OUT_DIR / fname, \"w\") as f:\n        f.write(\"date,tmin,tmax\\n\")\n        for day, lo, hi in records:\n            f.write(f\"{day},{lo},{hi}\\n\")\ncheck(\"[B-fetch] 92 Helsinki summer days downloaded\", len(helsinki) == 92)\ncheck(\"[B-fetch] 90 Nuorgam winter days downloaded\", len(nuorgam) == 90)\ncheck(\"[B-fetch] Helsinki record matches the archive (Jun 1: 8.1 to 16.3)\",\n      helsinki[0] == (\"2026-06-01\", 8.1, 16.3))\ncheck(\"[B-fetch] Nuorgam record matches the archive (Dec 1: -15.3 to -0.5)\",\n      nuorgam[0] == (\"2025-12-01\", -15.3, -0.5))"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-split]** Split the Helsinki days into a training set and a test set with a fixed seed; check the sizes."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "temps_hel = np.array([[lo, hi] for _, lo, hi in helsinki])\ntemps_nuo = np.array([[lo, hi] for _, lo, hi in nuorgam])\nrng = np.random.default_rng(42)\norder = rng.permutation(len(temps_hel))\ntrain, test = temps_hel[order[:62]], temps_hel[order[62:]]\ncheck(\"[B-split] 62 training and 30 test days\", len(train) == 62\n      and len(test) == 30)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-learn]** Learn the linear hypothesis by minimizing the average squared error on the training set; check that the fitted coefficients match the archive."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "A = np.stack([train[:, 0], np.ones(len(train))], 1)\nw1, w0 = np.linalg.lstsq(A, train[:, 1], rcond=None)[0]\nprint(f\"  learned hypothesis: tmax = {w1:.3f} * tmin + {w0:.3f}\")\ncheck(\"[B-learn] fitted coefficients match the archive\",\n      abs(w1 - 0.4806) < 5e-4 and abs(w0 - 14.5005) < 5e-4)\n\n\ndef avg_sq_err(points):\n    \"\"\"Average squared error of the learned hypothesis on (tmin, tmax).\"\"\"\n    return float(np.mean((w1 * points[:, 0] + w0 - points[:, 1]) ** 2))"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-shift]** Average squared error on the training set, the test set, and the Nuorgam winter days; check that the test error stays close to the training error while the Nuorgam error is several times larger, that the two feature ranges do not overlap, that the hypothesis overestimates every winter maximum, and that a line fitted to the winter days has a different slope (the conditional distribution shifted too)."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "err_train, err_test = avg_sq_err(train), avg_sq_err(test)\nerr_deploy = avg_sq_err(temps_nuo)\nprint(f\"  avg squared error: train {err_train:.2f}, test {err_test:.2f},\"\n      f\" Nuorgam winter {err_deploy:.2f}\")\ncheck(\"[B-shift] the test error is within a factor 2 of the training error\",\n      err_test < 2 * err_train)\ncheck(\"[B-shift] the Nuorgam error exceeds ten times the test error\",\n      err_deploy > 10 * err_test)\ncheck(\"[B-shift] the two feature ranges do not overlap\",\n      temps_nuo[:, 0].max() < train[:, 0].min())\ncheck(\"[B-shift] every winter maximum is overestimated\",\n      bool((w1 * temps_nuo[:, 0] + w0 > temps_nuo[:, 1]).all()))\ncheck(\"[B-shift] the mean winter day reads minimum -19.8, maximum -9.6\",\n      round(float(temps_nuo[:, 0].mean()), 1) == -19.8\n      and round(float(temps_nuo[:, 1].mean()), 1) == -9.6)\nw1_win, w0_win = np.linalg.lstsq(\n    np.stack([temps_nuo[:, 0], np.ones(len(temps_nuo))], 1),\n    temps_nuo[:, 1], rcond=None)[0]\nprint(f\"  line fitted to the winter days: tmax = {w1_win:.3f} * tmin\"\n      f\" + {w0_win:.3f}\")\ncheck(\"[B-shift] the winter days follow a different line (slope 0.766)\",\n      abs(w1_win - 0.766) < 2e-3)\n\n# ---- outputs: CSVs for the entry's pgfplots figure, preview PNG\nheader = \"tmin,tmax\"\nnp.savetxt(OUT_DIR / \"distshift_train.csv\", train, delimiter=\",\",\n           header=header, comments=\"\", fmt=\"%.1f\")\nnp.savetxt(OUT_DIR / \"distshift_test.csv\", test, delimiter=\",\",\n           header=header, comments=\"\", fmt=\"%.1f\")\nnp.savetxt(OUT_DIR / \"distshift_deploy.csv\", temps_nuo, delimiter=\",\",\n           header=header, comments=\"\", fmt=\"%.1f\")\nt_line = np.array([temps_nuo[:, 0].min() - 1.0, temps_hel[:, 0].max() + 1.0])\nnp.savetxt(OUT_DIR / \"distshift_hypothesis.csv\",\n           np.stack([t_line, w1 * t_line + w0], 1), delimiter=\",\",\n           header=header, comments=\"\", fmt=\"%.3f\")\nwith open(OUT_DIR / \"distshift_errors.csv\", \"w\") as f:\n    f.write(\"set,avg_sq_err\\n\")\n    f.write(f\"train,{err_train:.2f}\\ntest,{err_test:.2f}\\n\")\n    f.write(f\"deploy,{err_deploy:.2f}\\n\")\n\nfig, ax = plt.subplots(figsize=(7, 4.5))\nax.scatter(train[:, 0], train[:, 1], s=22, c=\"black\", marker=\"o\",\n           label=\"training set (Helsinki, summer)\")\nax.scatter(test[:, 0], test[:, 1], s=26, facecolors=\"none\",\n           edgecolors=\"black\", marker=\"s\",\n           label=\"test set (Helsinki, summer)\")\nax.scatter(temps_nuo[:, 0], temps_nuo[:, 1], s=26, facecolors=\"none\",\n           edgecolors=\"black\", marker=\"D\",\n           label=\"after deployment (Nuorgam, winter)\")\nax.plot(t_line, w1 * t_line + w0, \"k-\", lw=1.5,\n        label=\"learned hypothesis\")\nax.set_xlabel(\"daily minimum temperature (deg C)\")\nax.set_ylabel(\"daily maximum temperature (deg C)\")\nax.set_title(\"Learned from Helsinki summer days, deployed on Nuorgam winter days\\n\"\n             f\"avg squared error: train {err_train:.1f}, test {err_test:.1f},\"\n             f\" deployment {err_deploy:.1f}\")\nax.legend(frameon=False, fontsize=9)\nfig.tight_layout()\nfig.savefig(OUT_DIR / \"distshift.png\", dpi=120)\n\nfailed = [name for name, ok in report if not ok]\nprint(f\"{len(report) - len(failed)}/{len(report)} checks passed\"\n      + (f\"; FAILED: {failed}\" if failed else \"\"))\nraise SystemExit(1 if failed else 0)"
  }
 ]
}