{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3",
   "language": "python"
  },
  "language_info": {
   "name": "python"
  },
  "colab": {
   "name": "deepnet.ipynb"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# deep net \u2014 Python demo\n\nNumerical companion to the entry [deep net](https://dictionaryofml.org/terms/deepnet.html) of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org/): it recomputes what the entry states and prints one line per check.\n\nNumerical companion to the glossary entry 'deepnet' (deep net). One feature per data point, the day of the year, and one label, the daily maximum temperature at Krems (GeoSphere station 3805). The maximum temperature rises until July and falls again towards December, so no linear hypothesis of the day index predicts it: least squares on that single feature stays just below the training error of the constant hypothesis that always predicts the sample mean of the training labels.\n\nRequires NumPy and Matplotlib only, and uses fixed seeds, so the printed numbers reproduce exactly. Generated from [`pythondemos/deepnet.py`](https://dictionaryofml.org/terms/deepnet.py); CC BY 4.0.\n\nBefore running a downloaded copy, scan it: `uvx bandit -q -t B102,B307,B602,B605 deepnet.py` (B102,B307,B602,B605 are the exec, eval and shell checks of [bandit](https://bandit.readthedocs.io/)) reports nothing: the script runs no shell command and evaluates no generated code."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "# Notebook shim: the script resolves output paths relative to __file__,\n# which a notebook kernel does not define; everything lands in the\n# working directory instead.\nimport os\n__file__ = os.path.join(os.getcwd(), \"deepnet.py\")\nos.makedirs(\"pythondemos\", exist_ok=True)"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "\"\"\"A deep net as a composition of layers: the day of the year at Krems an\nder Donau turned into learned features, and the training and validation\nerror of the net against its depth.\n\nPurpose\n-------\nNumerical companion to the glossary entry 'deepnet' (deep net).  One\nfeature per data point, the day of the year, and one label, the daily\nmaximum temperature at Krems (GeoSphere station 3805).  The maximum\ntemperature rises until July and falls again towards December, so no\nlinear hypothesis of the day index predicts it: least squares on that\nsingle feature stays just below the training error of the constant\nhypothesis that always predicts the sample mean of the training labels.\n\nA deep net of depth L is built here exactly as the entry defines it:\nlayer l applies the affine map W^(l) a^(l-1) + b^(l) to the activations\nof the previous layer and then the rectified linear activation function\nto every entry of the result, with a^(0) the raw feature vector, and the\nlast layer applies an affine map alone, so that the net output is a\nlinear hypothesis of the activations a^(L-1).  Nothing but numpy is\nused: the forward pass is the composition, the gradient comes from\nbackpropagation (checked against a numerical gradient), and training\nis plain gradient descent with a fixed learning rate and a fixed seed.\n\nThe nets are trained on the 365 days of 2023 and validated on the 366\ndays of 2024.  Depth 1 is the linear hypothesis; depth 2 already cuts\nthe training error by a factor of four; depth 3 to 5 keep lowering the\ntraining error while the validation error rises again.  A separate net\nof depth 3 whose last hidden layer has two neurons shows the\nfeature-learning reading: its two activations are two learned features\nof the date, the net output is a linear hypothesis of them, and least\nsquares on those two features reaches almost the same training error as\nthe net's own output layer.\n\nWhat the depth costs is measured too.  A feature transformation chosen\nbeforehand, the sine and cosine of the date at the annual period and at\nhalf of it, needs five model parameters and reaches a lower validation\nerror than the depth-2 net with its 25.  And the training error of a net\nis not a convex function of its model parameters, so gradient descent\nreturns a different hypothesis from every initialization: five\ninitializations of the same depth-2 net are trained here and their\ntraining errors differ by more than a tenth.\n\nSelf-contained (numpy and matplotlib only), fixed seeds; the GeoSphere\ndownload falls back to the committed CSV.\n\nBlocks\n------\n[B-data]     Daily maximum temperature at Krems for 2023 and 2024 (731\n             days, no missing value), and the single feature, the day of\n             the year.\n[B-net]      The net as a composition of layers: forward pass, the\n             gradient by backpropagation checked against a numerical\n             gradient, and gradient descent.\n[B-depth]    Nets of depth 1 to 5 trained on the 2023 days; training\n             error and validation error against the depth.\n[B-curves]   The learned hypothesis maps of the depth-1, depth-2 and\n             depth-5 nets over the year, and the 20 days of 2023 drawn\n             in the entry's figure (every 18th day in calendar order).\n[B-features] A depth-3 net whose last hidden layer has two neurons: the\n             net output is a linear hypothesis of its two activations,\n             and least squares on those two learned features reaches\n             almost the same training error.\n[B-limits]   What the depth costs: least squares on a feature\n             transformation chosen beforehand, the sine and cosine of\n             the date, against the depth-2 net; and the spread of the\n             training error over five initializations of one net.\n[B-plot]     Preview: the hypothesis maps over the year, and the two\n             errors against the depth.\n\nOutputs\n-------\ndeepnet_weather.csv   : date, day of the year, maximum temperature for\n                        2023 and 2024 (committed copy of the download).\ndeepnet_depth.csv     : depth, training error, validation error.\ndeepnet_curves.csv    : day of the year, prediction of the depth-1,\n                        depth-2 and depth-5 net, in degrees Celsius.\ndeepnet_days_plot.csv : the 20 days of 2023 drawn in the entry's figure.\ndeepnet.png           : matplotlib preview (checking only).\n\"\"\"\n\n# /// script\n# requires-python = \">=3.10\"\n# dependencies = [\"matplotlib==3.10.7\", \"numpy==2.3.4\"]\n# ///\n\nimport datetime\nimport json\nimport urllib.request\nfrom pathlib import Path\n\nimport numpy as np\nimport matplotlib\n\nmatplotlib.use(\"Agg\")\nimport matplotlib.pyplot as plt\n\nOUT_DIR = Path(__file__).parent\n\nreport = []                         # collects (check name, pass/fail) pairs\n\n\ndef check(name, ok):                # records and prints one verification\n    report.append((name, bool(ok)))\n    print(f\"  [{'ok' if ok else 'FAIL'}] {name}\")"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-data]** Daily maximum temperature at Krems for 2023 and 2024 (731 days, no missing value), and the single feature, the day of the year."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-data] daily maximum temperature at Krems, 2023 and 2024\")\nURL = (\"https://dataset.api.hub.geosphere.at/v1/station/historical/\"\n       \"klima-v2-1d?parameters=tlmax&station_ids=3805\"\n       \"&start=2023-01-01&end=2024-12-31\")\nCSV = OUT_DIR / \"deepnet_weather.csv\"\n\n\ndef day_of_year(stamp):\n    return datetime.date(*map(int, stamp.split(\"-\"))).timetuple().tm_yday\n\n\ntry:\n    with urllib.request.urlopen(URL, timeout=120) as resp:\n        payload = json.load(resp)\n    series = payload[\"features\"][0][\"properties\"][\"parameters\"][\"tlmax\"]\n    records = [(stamp[:10], day_of_year(stamp[:10]), hi)\n               for stamp, hi in zip(payload[\"timestamps\"], series[\"data\"])]\n    with open(CSV, \"w\") as fh:\n        fh.write(\"date,doy,tmax\\n\")\n        for day, doy, hi in records:\n            fh.write(f\"{day},{doy},{hi}\\n\")\n    print(f\"    downloaded {len(records)} days\")\nexcept OSError as err:\n    print(f\"    download failed ({err}); reading the committed CSV\")\n    rows = CSV.read_text().strip().splitlines()[1:]\n    records = [(d, int(n), float(v))\n               for d, n, v in (r.split(\",\") for r in rows)]\n\ncheck(\"[B-data] 731 days for 2023 and 2024\", len(records) == 731)\ncheck(\"[B-data] every day carries a maximum temperature\",\n      all(hi is not None for _, _, hi in records))\ncheck(\"[B-data] the record matches the archive (1 July 2023: 24.9 deg, \"\n      \"day 182 of the year)\",\n      records[181][:2] == (\"2023-07-01\", 182)\n      and abs(records[181][2] - 24.9) < 1e-9)\n\ndates = [d for d, _, _ in records]\ndoy = np.array([n for _, n, _ in records], dtype=float)\ntmax = np.array([v for _, _, v in records], dtype=float)\nis23 = np.array([d.startswith(\"2023\") for d in dates])\n# the single feature of a day, the day of the year, centered on 2 July and\n# scaled so that the training feature values fill about [-1.7, 1.7]\nFEAT_OFFSET, FEAT_SCALE = 183.0, 105.0\nX = ((doy - FEAT_OFFSET) / FEAT_SCALE)[:, None]\n# the label is centered on the training sample mean, which leaves every\n# squared error unchanged and lets one learning rate serve every depth\nLABEL_MEAN = float(tmax[is23].mean())\ny = tmax - LABEL_MEAN\nX_train, y_train = X[is23], y[is23]\nX_val, y_val = X[~is23], y[~is23]\nerr_const = float((y_val ** 2).mean())\nprint(f\"    sample mean of the 2023 labels {LABEL_MEAN:.2f} degrees; the \"\n      f\"constant hypothesis has validation error {err_const:.1f}\")\ncheck(\"[B-data] 365 training days in 2023 and 366 validation days in 2024\",\n      len(y_train) == 365 and len(y_val) == 366)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-net]** The net as a composition of layers: forward pass, the gradient by backpropagation checked against a numerical gradient, and gradient descent."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-net] the net as a composition of layers; gradient by \"\n      \"backpropagation\")\n\n\ndef init(widths, seed):\n    \"\"\"Weight matrices and offset vectors of a net with the given widths.\"\"\"\n    rng = np.random.default_rng(seed)\n    mats = [rng.standard_normal((nxt, prv)) * np.sqrt(2.0 / prv)\n            for prv, nxt in zip(widths[:-1], widths[1:])]\n    offs = [rng.standard_normal(nxt) * 0.5 for nxt in widths[1:]]\n    return mats, offs\n\n\ndef forward(mats, offs, feats):\n    \"\"\"Activations of every layer: an affine map followed by the rectified\n    linear activation function, applied to every entry, and an affine map\n    alone in the last layer.\"\"\"\n    acts = [feats.T]\n    depth = len(mats)\n    for ell in range(depth):\n        pre = mats[ell] @ acts[-1] + offs[ell][:, None]\n        acts.append(np.maximum(pre, 0.0) if ell < depth - 1 else pre)\n    return acts\n\n\ndef predict(mats, offs, feats):\n    return forward(mats, offs, feats)[-1][0]\n\n\ndef average_loss(mats, offs, feats, labels):\n    \"\"\"Average squared error loss of the net on the given data points.\"\"\"\n    return float(((predict(mats, offs, feats) - labels) ** 2).mean())\n\n\ndef backprop(mats, offs, feats, labels):\n    \"\"\"The gradient of the average squared error loss with respect to\n    every weight matrix and offset vector, one layer at a time.\"\"\"\n    acts = forward(mats, offs, feats)\n    depth = len(mats)\n    g = (2.0 / len(labels)) * (acts[-1][0] - labels)[None, :]\n    d_mats, d_offs = [None] * depth, [None] * depth\n    for ell in range(depth - 1, -1, -1):\n        d_mats[ell] = g @ acts[ell].T\n        d_offs[ell] = g.sum(axis=1)\n        if ell > 0:\n            g = (mats[ell].T @ g) * (acts[ell] > 0)\n    return d_mats, d_offs\n\n\nLRATE, NRITER = 0.01, 20000\n\n\ndef train(widths, seed=1):\n    \"\"\"Gradient descent on the training error, from the fixed-seed start.\"\"\"\n    mats, offs = init(widths, seed)\n    for _ in range(NRITER):\n        d_mats, d_offs = backprop(mats, offs, X_train, y_train)\n        for ell in range(len(mats)):\n            mats[ell] -= LRATE * d_mats[ell]\n            offs[ell] -= LRATE * d_offs[ell]\n    return mats, offs\n\n\nmats0, offs0 = init([1, 8, 2, 1], 1)\nd_mats0, _ = backprop(mats0, offs0, X_train, y_train)\nEPS = 1e-6\nrel = 0.0\nrng_check = np.random.default_rng(7)\nfor ell in range(len(mats0)):\n    for _ in range(5):\n        i = int(rng_check.integers(mats0[ell].shape[0]))\n        j = int(rng_check.integers(mats0[ell].shape[1]))\n        mats0[ell][i, j] += EPS\n        up = average_loss(mats0, offs0, X_train, y_train)\n        mats0[ell][i, j] -= 2 * EPS\n        down = average_loss(mats0, offs0, X_train, y_train)\n        mats0[ell][i, j] += EPS\n        numeric = (up - down) / (2 * EPS)\n        rel = max(rel, abs(numeric - d_mats0[ell][i, j]) / max(1.0, abs(numeric)))\nprint(f\"    largest relative deviation over 15 weights: {rel:.2e}\")\ncheck(\"[B-net] backpropagation agrees with the numerical gradient\",\n      rel < 1e-6)\none = np.array([[0.4]])\nacts_one = forward(*init([1, 8, 2, 1], 1), one)\ncheck(\"[B-net] every hidden activation is nonnegative, the last layer is \"\n      \"affine only\",\n      all((a >= 0).all() for a in acts_one[1:-1]) and acts_one[-1].shape == (1, 1))"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-depth]** Nets of depth 1 to 5 trained on the 2023 days; training error and validation error against the depth."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-depth] nets of depth 1 to 5 trained on the 2023 days\")\nWIDTH, DEPTHS = 8, [1, 2, 3, 4, 5]\nnets, err_train, err_val = {}, {}, {}\nfor depth in DEPTHS:\n    widths = [1] + [WIDTH] * (depth - 1) + [1]\n    nets[depth] = train(widths)\n    err_train[depth] = average_loss(*nets[depth], X_train, y_train)\n    err_val[depth] = average_loss(*nets[depth], X_val, y_val)\n    print(f\"    depth {depth} (widths {widths}): training error \"\n          f\"{err_train[depth]:.1f}, validation error {err_val[depth]:.1f}\")\nwith open(OUT_DIR / \"deepnet_depth.csv\", \"w\") as fh:\n    fh.write(\"depth,trainerr,valerr\\n\")\n    for depth in DEPTHS:\n        fh.write(f\"{depth},{err_train[depth]:.2f},{err_val[depth]:.2f}\\n\")\ncheck(\"[B-depth] the depth-1 net is a linear hypothesis and stays within \"\n      \"6 percent of the constant hypothesis on the training days\",\n      abs(err_train[1] - float((y_train ** 2).mean())) / err_train[1] < 0.06)\ncheck(\"[B-depth] one hidden layer cuts the training error by at least a \"\n      \"factor of four\", err_train[1] > 4 * err_train[2])\ncheck(\"[B-depth] the training error falls with every added layer\",\n      all(err_train[a] > err_train[b] for a, b in zip(DEPTHS, DEPTHS[1:])))\ncheck(\"[B-depth] the validation error is smallest at depth 2 and rises \"\n      \"with every further layer\",\n      min(err_val, key=err_val.get) == 2\n      and all(err_val[a] < err_val[b] for a, b in zip(DEPTHS[1:], DEPTHS[2:])))"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-curves]** The learned hypothesis maps of the depth-1, depth-2 and depth-5 nets over the year, and the 20 days of 2023 drawn in the entry's figure (every 18th day in calendar order)."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-curves] the learned hypothesis maps over the year\")\ngrid_doy = np.arange(1, 366, dtype=float)\ngrid_X = ((grid_doy - FEAT_OFFSET) / FEAT_SCALE)[:, None]\ncurves = {d: predict(*nets[d], grid_X) + LABEL_MEAN for d in (1, 2, 5)}\nwith open(OUT_DIR / \"deepnet_curves.csv\", \"w\") as fh:\n    fh.write(\"doy,h1,h2,h5\\n\")\n    for k, n in enumerate(grid_doy):\n        fh.write(f\"{int(n)},{curves[1][k]:.3f},{curves[2][k]:.3f},\"\n                 f\"{curves[5][k]:.3f}\\n\")\ncheck(\"[B-curves] the depth-1 hypothesis is a straight line in the day \"\n      \"index (second differences vanish)\",\n      np.allclose(np.diff(curves[1], 2), 0.0, atol=1e-9))\npeak2 = int(grid_doy[int(np.argmax(curves[2]))])\nprint(f\"    the depth-2 hypothesis peaks on day {peak2} at \"\n      f\"{curves[2].max():.1f} degrees, and ranges over \"\n      f\"{curves[2].max() - curves[2].min():.1f} degrees; the depth-1 \"\n      f\"hypothesis ranges over {curves[1].max() - curves[1].min():.1f}\")\ncheck(\"[B-curves] the depth-2 hypothesis peaks in July or August\",\n      182 <= peak2 <= 243)\n# the entry's figure draws 20 of the 365 training days, every 18th day\n# in calendar order, so that the single points stay legible\ndrawn = np.flatnonzero(is23)[::18][:20]\nwith open(OUT_DIR / \"deepnet_days_plot.csv\", \"w\") as fh:\n    fh.write(\"doy,tmax\\n\")\n    for k in drawn:\n        fh.write(f\"{int(doy[k])},{tmax[k]:.1f}\\n\")\ncheck(\"[B-curves] 20 days drawn, every 18th day of 2023 from 1 January\",\n      len(drawn) == 20 and doy[drawn[0]] == 1 and doy[drawn[1]] == 19)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-features]** A depth-3 net whose last hidden layer has two neurons: the net output is a linear hypothesis of its two activations, and least squares on those two learned features reaches almost the same training error."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-features] a depth-3 net whose last hidden layer has two neurons\")\nmats_f, offs_f = train([1, WIDTH, 2, 1], seed=2)\nZ_train = forward(mats_f, offs_f, X_train)[-2].T        # 365 x 2\nZ_val = forward(mats_f, offs_f, X_val)[-2].T\nout_train = predict(mats_f, offs_f, X_train)\nlinear_on_Z = Z_train @ mats_f[-1][0] + offs_f[-1][0]\ncheck(\"[B-features] the net output is the linear hypothesis of the two \"\n      \"activations given by the last layer\",\n      np.allclose(out_train, linear_on_Z))\ncheck(\"[B-features] both learned features vary over the training days\",\n      (Z_train.std(axis=0) > 1e-6).all())\nA_train = np.hstack([Z_train, np.ones((len(Z_train), 1))])\nw_ls = np.linalg.lstsq(A_train, y_train, rcond=None)[0]\nerr_net = average_loss(mats_f, offs_f, X_train, y_train)\nerr_ls = float(((A_train @ w_ls - y_train) ** 2).mean())\nA_val = np.hstack([Z_val, np.ones((len(Z_val), 1))])\nerr_ls_val = float(((A_val @ w_ls - y_val) ** 2).mean())\nraw = np.hstack([X_train, np.ones((len(X_train), 1))])\nw_raw = np.linalg.lstsq(raw, y_train, rcond=None)[0]\nerr_raw = float(((raw @ w_raw - y_train) ** 2).mean())\nprint(f\"    training error of the net {err_net:.2f}, of least squares on \"\n      f\"its two learned features {err_ls:.2f}, of least squares on the raw \"\n      f\"feature {err_raw:.2f}\")\nprint(f\"    validation error of the net \"\n      f\"{average_loss(mats_f, offs_f, X_val, y_val):.2f}, of least squares \"\n      f\"on the two learned features {err_ls_val:.2f}\")\ncheck(\"[B-features] least squares on the two learned features cannot beat \"\n      \"the net's own output layer by more than 1 percent\",\n      err_ls <= err_net and err_net - err_ls < 0.01 * err_net)\ncheck(\"[B-features] the two learned features carry most of what the raw \"\n      \"feature does not: least squares on them has less than a third of \"\n      \"the training error\", err_ls < err_raw / 3)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-limits]** What the depth costs: least squares on a feature transformation chosen beforehand, the sine and cosine of the date, against the depth-2 net; and the spread of the training error over five initializations of one net."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "# ---- spread over initializations\nprint(\"[B-limits] least squares on the sine and cosine of the date, and \"\n      \"five initializations of the depth-2 net\")\nPERIOD = 2.0 * np.pi / 365.0\n\n\ndef chosen_features(days):\n    \"\"\"The feature transformation chosen beforehand: the sine and cosine of\n    the date at the annual period and at half of it, and a constant.\"\"\"\n    return np.column_stack([np.sin(PERIOD * days), np.cos(PERIOD * days),\n                            np.sin(2 * PERIOD * days),\n                            np.cos(2 * PERIOD * days),\n                            np.ones(len(days))])\n\n\nC_train, C_val = chosen_features(doy[is23]), chosen_features(doy[~is23])\nw_chosen = np.linalg.lstsq(C_train, y_train, rcond=None)[0]\nerr_chosen_train = float(((C_train @ w_chosen - y_train) ** 2).mean())\nerr_chosen_val = float(((C_val @ w_chosen - y_val) ** 2).mean())\nnr_param_net2 = sum(m.size for m in nets[2][0]) + sum(\n    o.size for o in nets[2][1])\nprint(f\"    chosen transformation ({w_chosen.size} model parameters): \"\n      f\"training error {err_chosen_train:.1f}, validation error \"\n      f\"{err_chosen_val:.1f}\")\nprint(f\"    depth-2 net ({nr_param_net2} model parameters): training error \"\n      f\"{err_train[2]:.1f}, validation error {err_val[2]:.1f}\")\ncheck(\"[B-limits] the chosen transformation has a lower validation error \"\n      \"than the depth-2 net, with fewer model parameters\",\n      err_chosen_val < err_val[2] and w_chosen.size < nr_param_net2)\ncheck(\"[B-limits] the depth-2 net still has the lower training error, so \"\n      \"the chosen transformation is not simply the larger model\",\n      err_train[2] < err_chosen_train)\nraw_val = np.hstack([X_val, np.ones((len(X_val), 1))])\nerr_raw_val = float(((raw_val @ w_raw - y_val) ** 2).mean())\nprint(f\"    least squares on the raw feature (2 model parameters): \"\n      f\"training error {err_raw:.1f}, validation error {err_raw_val:.1f}\")\ncheck(\"[B-limits] least squares on the raw feature has the validation \"\n      \"error of the depth-1 net, which is the same linear hypothesis\",\n      abs(err_raw_val - err_val[1]) < 0.05)\n\nSEEDS = [1, 2, 3, 4, 5]\nseed_train, seed_val = [], []\nfor seed in SEEDS:\n    mats_s, offs_s = train([1, WIDTH, 1], seed=seed)\n    seed_train.append(average_loss(mats_s, offs_s, X_train, y_train))\n    seed_val.append(average_loss(mats_s, offs_s, X_val, y_val))\nprint(\"    five initializations of the depth-2 net, training error: \"\n      + \", \".join(f\"{v:.1f}\" for v in seed_train))\nprint(\"    the same five, validation error: \"\n      + \", \".join(f\"{v:.1f}\" for v in seed_val))\ncheck(\"[B-limits] the five initializations give five different training \"\n      \"errors\", len({round(v, 6) for v in seed_train}) == 5)\ncheck(\"[B-limits] the largest of the five training errors exceeds the \"\n      \"smallest by more than a tenth\",\n      max(seed_train) > 1.1 * min(seed_train))\ncheck(\"[B-limits] four of the five end above the smallest training error \"\n      \"the five reach, so a single run cannot be relied on to return the \"\n      \"best hypothesis of the net\",\n      sum(v > min(seed_train) for v in seed_train) == 4)\ncheck(\"[B-limits] the first of the five repeats the depth-2 net of the \"\n      \"depth block, which starts from the same initialization\",\n      abs(seed_train[0] - err_train[2]) < 1e-9)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-plot]** Preview: the hypothesis maps over the year, and the two errors against the depth."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-plot] preview\")\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 4))\nax1.plot(grid_doy, curves[1], \"k--\", label=\"depth 1 (dashed line)\")\nax1.plot(grid_doy, curves[2], \"k-\", label=\"depth 2 (solid curve)\")\nax1.plot(grid_doy, curves[5], \"k:\", label=\"depth 5 (dotted curve)\")\nax1.plot(doy[drawn], tmax[drawn], \"ko\", ms=5,\n         label=\"20 of the 365 days of 2023 (filled circles)\")\nax1.set_xlabel(\"day of the year\")\nax1.set_ylabel(\"maximum temperature in degrees Celsius\")\nax1.set_title(\"Learned hypothesis maps at Krems, 2023\")\nax1.legend(frameon=False, fontsize=8)\nax2.plot(DEPTHS, [err_train[d] for d in DEPTHS], \"k-o\", ms=5,\n         label=\"training error (solid curve, filled circles)\")\nax2.plot(DEPTHS, [err_val[d] for d in DEPTHS], \"k--s\", ms=6,\n         markerfacecolor=\"none\", label=\"validation error (dashed curve, \"\n         \"open squares)\")\nax2.set_xlabel(\"depth of the net\")\nax2.set_ylabel(\"average squared error loss in squared degrees\")\nax2.set_title(\"Depth lowers the training error, not the validation error\")\nax2.set_xticks(DEPTHS)\nax2.legend(frameon=False, fontsize=8)\nfig.tight_layout()\nfig.savefig(OUT_DIR / \"deepnet.png\", dpi=110)\n\nprint()\nfailed = [n for n, ok in report if not ok]\nprint(f\"{len(report) - len(failed)}/{len(report)} checks passed\"\n      + (f\"; FAILED: {failed}\" if failed else \"\"))"
  }
 ]
}