{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3",
   "language": "python"
  },
  "language_info": {
   "name": "python"
  },
  "colab": {
   "name": "ann.ipynb"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# artificial neural network \u2014 Python demo\n\nNumerical companion to the entry [artificial neural network](https://dictionaryofml.org/terms/ann.html) of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org/): it recomputes what the entry states and prints one line per check.\n\nNumerical companion to the glossary entry 'ann' (artificial neural network). Every quantity is computed twice -- once by the network, once by hand -- so each claim of the entry can be checked: the artificial neuron applies an activation function to the weighted sum of its inputs plus an offset term; the five-node network delivers its prediction from the inputs x1, a1, a2; a deep net without skip connections equals the concatenation of its layer-wise feature transformations; adding a skip connection or tuning a single edge weight changes the hypothesis the network represents.\n\nRequires NumPy and Matplotlib only, and uses fixed seeds, so the printed numbers reproduce exactly. Generated from [`pythondemos/ann.py`](https://dictionaryofml.org/terms/ann.py); CC BY 4.0."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "# Notebook shim: the script resolves output paths relative to __file__,\n# which a notebook kernel does not define; everything lands in the\n# working directory instead.\nimport os\n__file__ = os.path.join(os.getcwd(), \"ann.py\")\nos.makedirs(\"pythondemos\", exist_ok=True)"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "\"\"\"The ANN of the entry's figures, computed number by number: a single\nartificial neuron, the five-node network, and a deep net evaluated as a\nconcatenation of layer-wise feature transformations.\n\nPurpose\n-------\nNumerical companion to the glossary entry 'ann' (artificial neural\nnetwork).  Every quantity is computed twice -- once by the network,\nonce by hand -- so each claim of the entry can be checked: the\nartificial neuron applies an activation function to the weighted sum of\nits inputs plus an offset term; the five-node network delivers its\nprediction from the inputs x1, a1, a2; a deep net without skip\nconnections equals the concatenation of its layer-wise feature\ntransformations; adding a skip connection or tuning a single edge\nweight changes the hypothesis the network represents.\n\nAll edge weights and offsets are fixed constants stated in the code\n(no randomness), and the activation function is sigma(z) = max(0, z).\n\nBlocks\n------\n[B-neuron] A single artificial neuron: output a1 = sigma(w1 x1 + w2 x2\n           + b), checked against the hand-computed value.\n[B-dag]    The five-node network: hidden outputs a1, a2 feed the output\n           node together with x1; the prediction is checked against the\n           hand-computed weighted sum.\n[B-layers] A deep net with two hidden layers of three neurons each:\n           evaluating it layer by layer equals the concatenation\n           Phi3(Phi2(Phi1(x))) of the layer-wise feature\n           transformations, starting from the feature vector\n           x = (x1, x2, x3).\n[B-skip]   A skip connection from the first hidden layer straight to\n           the output changes the prediction.\n[B-tune]   The edge weights are tunable model parameters: changing one\n           of them changes the hypothesis, shown as two prediction\n           surfaces over the inputs (x1, x2) of the five-node network.\n\nOutputs\n-------\nann_surface.csv       : x1, x2, h -- prediction of the five-node network\nann_surface_tuned.csv : the same after changing the edge weight w5\nann.png               : preview (checking only) -- the neuron's output\n                        for two offsets, and the two prediction surfaces\n\"\"\"\n\nfrom pathlib import Path\n\nimport numpy as np\nimport matplotlib\n\nmatplotlib.use(\"Agg\")\nimport matplotlib.pyplot as plt\n\nOUT_DIR = Path(__file__).parent\n\nreport = []                         # collects (check name, pass/fail) pairs\n\n\ndef check(name, ok):                # records and prints one verification\n    report.append((name, bool(ok)))\n    print(f\"  [{'ok' if ok else 'FAIL'}] {name}\")\n\n\ndef actfun(z):\n    \"\"\"The activation function sigma(z) = max(0, z).\"\"\"\n    return np.maximum(0.0, z)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-neuron]** A single artificial neuron: output a1 = sigma(w1 x1 + w2 x2 + b), checked against the hand-computed value."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "w1, w2, b = 0.8, -0.4, 0.3          # edge weights and offset term\nx1, x2 = 1.0, 2.0\na1 = actfun(w1 * x1 + w2 * x2 + b)\ncheck(\"[B-neuron] a1 = sigma(0.8*1.0 - 0.4*2.0 + 0.3) = 0.3\",\n      abs(a1 - 0.3) < 1e-12)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-dag]** The five-node network: hidden outputs a1, a2 feed the output node together with x1; the prediction is checked against the hand-computed weighted sum."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "W_DAG = dict(w1=0.8, w2=-0.4, w3=0.5, w4=0.9, w5=1.2, w6=-0.7, w7=0.4)\nB1, B2 = 0.3, 0.0                   # offset terms of the neurons a1, a2\n\n\ndef dag_predict(x1, x2, w):\n    \"\"\"Prediction of the five-node network for the features x1, x2.\"\"\"\n    a1 = actfun(w[\"w1\"] * x1 + w[\"w2\"] * x2 + B1)\n    a2 = actfun(w[\"w3\"] * x1 + w[\"w4\"] * x2 + B2)\n    return w[\"w5\"] * a1 + w[\"w6\"] * a2 + w[\"w7\"] * x1\n\n\nh = dag_predict(1.0, 2.0, W_DAG)\ncheck(\"[B-dag] the prediction is the weighted sum of x1, a1, a2\",\n      abs(h - (1.2 * 0.3 - 0.7 * 2.3 + 0.4 * 1.0)) < 1e-12)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-layers]** A deep net with two hidden layers of three neurons each: evaluating it layer by layer equals the concatenation Phi3(Phi2(Phi1(x))) of the layer-wise feature transformations, starting from the feature vector x = (x1, x2, x3)."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "W1 = np.array([[0.8, -0.4, 0.2], [0.5, 0.9, -0.3], [-0.6, 0.7, 0.4]])\nO1 = np.array([0.3, 0.0, 0.1])      # offset terms of the first hidden layer\nW2 = np.array([[0.7, -0.5, 0.3], [0.2, 0.6, -0.4], [0.5, 0.1, 0.8]])\nO2 = np.array([0.0, 0.2, -0.1])     # offset terms of the second hidden layer\nW3 = np.array([1.2, -0.7, 0.4])\nO3 = 0.5                            # offset term of the output neuron\n\n\ndef phi1(v):                        # first hidden layer, R^3 -> R^3\n    return actfun(W1 @ v + O1)\n\n\ndef phi2(v):                        # second hidden layer, R^3 -> R^3\n    return actfun(W2 @ v + O2)\n\n\ndef phi3(v):                        # output layer, R^3 -> R\n    return float(W3 @ v + O3)\n\n\nx = np.array([1.0, 2.0, -1.0])      # feature vector x = (x1, x2, x3)\nnested = phi3(phi2(phi1(x)))\nvalue = x.copy()                    # the same net, one layer after the other\nfor layer in (phi1, phi2):\n    value = layer(value)\nsequential = phi3(value)\ncheck(\"[B-layers] layer-by-layer evaluation equals Phi3(Phi2(Phi1(x)))\",\n      abs(nested - sequential) < 1e-12)\ncheck(\"[B-layers] hidden layers map R^3 -> R^3, the output layer to a \"\n      \"single prediction\",\n      phi1(x).shape == (3,) and phi2(phi1(x)).shape == (3,)\n      and np.isscalar(nested))\nprint(f\"  Phi1(x) = {phi1(x)},  prediction = {nested:+.3f}\")"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-skip]** A skip connection from the first hidden layer straight to the output changes the prediction."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "W_SKIP = 1.0                        # edge weight of the skip connection\nh_skip = nested + W_SKIP * phi1(x)[0]\ncheck(\"[B-skip] adding the skip connection changes the prediction\",\n      abs(h_skip - nested) > 1e-9)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-tune]** The edge weights are tunable model parameters: changing one of them changes the hypothesis, shown as two prediction surfaces over the inputs (x1, x2) of the five-node network."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "W_TUNED = dict(W_DAG, w5=-1.2)\ncheck(\"[B-tune] changing the edge weight w5 changes the prediction\",\n      abs(dag_predict(1.0, 2.0, W_TUNED) - h) > 1e-9)\n\ng = np.linspace(-2.0, 2.0, 41)\nX1, X2 = np.meshgrid(g, g)\nH = dag_predict(X1, X2, W_DAG)\nH_TUNED = dag_predict(X1, X2, W_TUNED)\nfor fname, surface in ((\"ann_surface.csv\", H),\n                       (\"ann_surface_tuned.csv\", H_TUNED)):\n    rows = np.stack([X1.ravel(), X2.ravel(), surface.ravel()], 1)\n    np.savetxt(OUT_DIR / fname, rows, delimiter=\",\",\n               header=\"x1,x2,h\", comments=\"\", fmt=\"%.4f\")\n\nfig, axes = plt.subplots(1, 3, figsize=(11, 3.4))\nax = axes[0]\nz1 = np.linspace(-2.0, 2.0, 200)\nfor offset, style in ((0.3, \"-\"), (-0.3, \"--\")):\n    ax.plot(z1, actfun(w1 * z1 + w2 * x2 + offset), style, color=\"black\",\n            label=f\"offset $b = {offset}$\")\nax.set_xlabel(\"feature $x_1$ (with $x_2 = 2$ fixed)\")\nax.set_ylabel(\"neuron output $a_1$\")\nax.set_title(\"a single artificial neuron\")\nax.legend(frameon=False)\n\nfor ax, surface, title in ((axes[1], H, \"five-node network\"),\n                           (axes[2], H_TUNED,\n                            \"after tuning edge weight $w_5$\")):\n    filled = ax.contourf(X1, X2, surface, levels=12, cmap=\"Greys\")\n    lines = ax.contour(X1, X2, surface, levels=6, colors=\"black\",\n                       linewidths=0.6)\n    ax.clabel(lines, fontsize=6)\n    ax.set_xlabel(\"feature $x_1$\")\n    ax.set_ylabel(\"feature $x_2$\")\n    ax.set_title(f\"prediction $h(x_1, x_2)$\\n{title}\")\n\nfig.tight_layout()\nfig.savefig(OUT_DIR / \"ann.png\", dpi=150)\n\nfailed = [name for name, ok in report if not ok]\nprint(f\"{len(report) - len(failed)}/{len(report)} checks passed\"\n      + (f\", FAILED: {failed}\" if failed else \"\"))"
  }
 ]
}