{
"nbformat": 4,
"nbformat_minor": 5,
"metadata": {
"kernelspec": {
"name": "python3",
"display_name": "Python 3",
"language": "python"
},
"language_info": {
"name": "python"
},
"colab": {
"name": "tabulardata.ipynb"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": "# tabular data \u2014 Python demo\n\nNumerical companion to the entry [tabular data](https://dictionaryofml.org/terms/tabulardata.html) of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org/): it recomputes what the entry states and prints one line per check.\n\nThe demo gathers actual weather measurements from the open data server of the Finnish Meteorological Institute (FMI): daily minimum temperature, precipitation, and maximum temperature at the Helsinki Kaisaniemi station (fmisid 100971) for the year 2024. The measurements arrive as one value per (day, attribute) pair and are stored as a table: one row per day, one column per attribute.\n\nRequires NumPy and Matplotlib only, and uses fixed seeds, so the printed numbers reproduce exactly. Generated from [`pythondemos/tabulardata.py`](https://dictionaryofml.org/terms/tabulardata.py); CC BY 4.0."
},
{
"cell_type": "code",
"metadata": {},
"execution_count": null,
"outputs": [],
"source": "# Notebook shim: the script resolves output paths relative to __file__,\n# which a notebook kernel does not define; everything lands in the\n# working directory instead.\nimport os\n__file__ = os.path.join(os.getcwd(), \"tabulardata.py\")\nos.makedirs(\"pythondemos\", exist_ok=True)"
},
{
"cell_type": "code",
"metadata": {},
"execution_count": null,
"outputs": [],
"source": "\"\"\"tabulardata.py \u2014 numerical companion to the glossary entry\n'tabulardata' (tabular data).\n\nThe demo gathers actual weather measurements from the open data server\nof the Finnish Meteorological Institute (FMI): daily minimum\ntemperature, precipitation, and maximum temperature at the Helsinki\nKaisaniemi station (fmisid 100971) for the year 2024. The\nmeasurements arrive as one value per (day, attribute) pair and are\nstored as a table: one row per day, one column per attribute.\n\nThe blocks verify the entry's claims: every row has one cell per\ncolumn and the cells of one column hold values of the same attribute;\nthe value range of a cell can include a special value (FMI reports the\nprecipitation of a dry day as -1.0); and which attributes serve as the\nfeature and which as the label is a design choice \u2014 here the minimum\ntemperature column is read as the feature and the maximum temperature\ncolumn as the label of a learning task.\n\nDeterministic: historical measurements, no randomness.\nSelf-contained: numpy + matplotlib only (stdlib urllib for the\ndownload).\n\nBlocks\n------\n[B-fetch] Download the 366 daily rows for 2024 from the FMI open\n data server and write them to tabulardata_weather.csv;\n check the count, one pinned measurement, and that the\n table has no empty cell \u2014 dry days carry the special\n precipitation value -1.0 instead.\n[B-table] Select four consecutive days that contain both genuine\n precipitation values and the special value -1.0; this\n window is the table shown in the entry's figure\n (tabulardata_table.csv).\n[B-picture] Read the minimum-temperature column as the feature and\n the maximum-temperature column as the label: scatterplot\n of the 366 days and a fitted curve (degree-3 least\n squares), written to tabulardata_scatter.csv and\n tabulardata_curve.csv for the entry's figure.\n\nOutputs\n-------\ntabulardata_weather.csv : day, tmin, rrday, tmax for the 366 days\ntabulardata_table.csv : the four rows shown in the entry's figure\ntabulardata_scatter.csv : tmin, tmax of the 366 days\ntabulardata_curve.csv : tmin, pred -- the fitted curve on a grid\ntabulardata.png : preview (checking only) -- the scatterplot\n with the curve, and the four-day table\n\"\"\"\n\nimport re\nimport urllib.request\nfrom pathlib import Path\n\nimport numpy as np\nimport matplotlib\n\nmatplotlib.use(\"Agg\")\nimport matplotlib.pyplot as plt\n\nOUT_DIR = Path(__file__).parent\n\nreport = []\n\n\ndef check(name, ok):\n report.append((name, bool(ok)))\n print(f\" [{'ok' if ok else 'FAIL'}] {name}\")"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "**[B-fetch]** Download the 366 daily rows for 2024 from the FMI open data server and write them to tabulardata_weather.csv; check the count, one pinned measurement, and that the table has no empty cell \u2014 dry days carry the special precipitation value -1.0 instead."
},
{
"cell_type": "code",
"metadata": {},
"execution_count": null,
"outputs": [],
"source": "URL = (\"https://opendata.fmi.fi/wfs?service=WFS&version=2.0.0\"\n \"&request=getFeature\"\n \"&storedquery_id=fmi::observations::weather::daily::simple\"\n \"&fmisid=100971&starttime=2024-01-01T00:00:00Z\"\n \"&endtime=2024-12-31T00:00:00Z¶meters=tmin,tmax,rrday\")\nwith urllib.request.urlopen(URL, timeout=120) as resp:\n xml = resp.read().decode()\ntriples = re.findall(\n r\"(\\S+?)T.*?\\s*\"\n r\"(\\w+)\\s*\"\n r\"(\\S+)\", xml, re.S)\nrows = {}\nfor day, name, value in triples:\n rows.setdefault(day, {})[name] = float(value)\ndays = sorted(rows)\nwith open(OUT_DIR / \"tabulardata_weather.csv\", \"w\") as f:\n f.write(\"day,tmin,rrday,tmax\\n\")\n for d in days:\n r = rows[d]\n f.write(f\"{d},{r['tmin']:.1f},{r['rrday']:.1f},{r['tmax']:.1f}\\n\")\ncheck(\"[B-fetch] 366 daily rows downloaded for 2024\", len(days) == 366)\ncheck(\"[B-fetch] the record matches the archive \"\n \"(May 2: 4.5 to 14.5 degrees)\",\n rows[\"2024-05-02\"][\"tmin\"] == 4.5\n and rows[\"2024-05-02\"][\"tmax\"] == 14.5)\ncheck(\"[B-fetch] every row has one cell per column (no empty cells)\",\n all(len(rows[d]) == 3 for d in days))\nn_special = sum(rows[d][\"rrday\"] == -1.0 for d in days)\nprint(f\" special precipitation value -1.0 (dry day) on \"\n f\"{n_special} of {len(days)} days\")\ncheck(\"[B-fetch] the value range of the precipitation column includes \"\n \"the special value -1.0 for a dry day\",\n 0 < n_special < len(days))"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "**[B-table]** Select four consecutive days that contain both genuine precipitation values and the special value -1.0; this window is the table shown in the entry's figure (tabulardata_table.csv)."
},
{
"cell_type": "code",
"metadata": {},
"execution_count": null,
"outputs": [],
"source": "window = None\nfor k in range(len(days) - 3):\n rr = [rows[d][\"rrday\"] for d in days[k:k + 4]]\n if sum(v > 0 for v in rr) >= 2 and sum(v == -1.0 for v in rr) >= 1:\n window = days[k:k + 4]\n break\nprint(f\" four-day window for the figure: {window[0]} .. {window[-1]}\")\nwith open(OUT_DIR / \"tabulardata_table.csv\", \"w\") as f:\n f.write(\"day,tmin,rrday,tmax\\n\")\n for d in window:\n r = rows[d]\n f.write(f\"{d},{r['tmin']:.1f},{r['rrday']:.1f},{r['tmax']:.1f}\\n\")\ncheck(\"[B-table] the window mixes genuine precipitation values with \"\n \"the special value\",\n any(rows[d][\"rrday\"] > 0 for d in window)\n and any(rows[d][\"rrday\"] == -1.0 for d in window))\ncheck(\"[B-table] all four rows share the same fixed set of attributes\",\n all(sorted(rows[d]) == [\"rrday\", \"tmax\", \"tmin\"] for d in window))"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "**[B-picture]** Read the minimum-temperature column as the feature and the maximum-temperature column as the label: scatterplot of the 366 days and a fitted curve (degree-3 least squares), written to tabulardata_scatter.csv and tabulardata_curve.csv for the entry's figure."
},
{
"cell_type": "code",
"metadata": {},
"execution_count": null,
"outputs": [],
"source": "tmin = np.array([rows[d][\"tmin\"] for d in days])\ntmax = np.array([rows[d][\"tmax\"] for d in days])\nnp.savetxt(OUT_DIR / \"tabulardata_scatter.csv\",\n np.stack([tmin, tmax], 1),\n delimiter=\",\", header=\"tmin,tmax\", comments=\"\", fmt=\"%.1f\")\ncoef = np.polyfit(tmin, tmax, 3)\ngrid = np.linspace(tmin.min(), tmin.max(), 200)\nnp.savetxt(OUT_DIR / \"tabulardata_curve.csv\",\n np.stack([grid, np.polyval(coef, grid)], 1),\n delimiter=\",\", header=\"tmin,pred\", comments=\"\", fmt=\"%.2f\")\nerr_fit = float(np.mean((tmax - np.polyval(coef, tmin)) ** 2))\nerr_const = float(np.var(tmax))\nprint(f\" average squared error: fitted curve {err_fit:.1f}, \"\n f\"constant {err_const:.1f}\")\ncheck(\"[B-picture] the fitted curve predicts the label far better \"\n \"than a constant\", err_fit < err_const / 3)\ncheck(\"[B-picture] warmer mornings go with warmer days \"\n \"(positive correlation)\",\n float(np.corrcoef(tmin, tmax)[0, 1]) > 0.8)\n\n# ---- preview\nfig, axes = plt.subplots(1, 2, figsize=(9.6, 3.6),\n gridspec_kw={\"width_ratios\": [3, 2]})\nax = axes[0]\nax.plot(tmin, tmax, \"o\", ms=2.5, color=\"0.6\", label=\"days of 2024\")\nax.plot(grid, np.polyval(coef, grid), \"-\", lw=2, color=\"black\",\n label=\"fitted curve\")\nax.set_xlabel(\"minimum temperature of the day (\u00b0C)\")\nax.set_ylabel(\"maximum temperature of the day (\u00b0C)\")\nax.set_title(\"feature: min. temp., label: max. temp. (Kaisaniemi)\")\nax.legend(frameon=False, fontsize=8)\nax = axes[1]\nax.axis(\"off\")\ncells = [[d, f\"{rows[d]['tmin']:.1f}\", f\"{rows[d]['rrday']:.1f}\",\n f\"{rows[d]['tmax']:.1f}\"] for d in window]\ntab = ax.table(cellText=cells,\n colLabels=[\"day\", \"min. temp.\", \"precip.\", \"max. temp.\"],\n loc=\"center\")\ntab.auto_set_font_size(False)\ntab.set_fontsize(8)\nax.set_title(\"four rows of the table (-1.0 = dry day)\")\nfig.tight_layout()\nfig.savefig(OUT_DIR / \"tabulardata.png\", dpi=150)\n\nfailed = [name for name, ok in report if not ok]\nprint(f\"{len(report) - len(failed)}/{len(report)} checks passed\"\n + (f\", FAILED: {failed}\" if failed else \"\"))"
}
]
}