{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3",
   "language": "python"
  },
  "language_info": {
   "name": "python"
  },
  "colab": {
   "name": "interpretability.ipynb"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# interpretability \u2014 Python demo\n\nNumerical companion to the entry [interpretability](https://dictionaryofml.org/terms/interpretability.html) of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org/): it recomputes what the entry states and prints one line per check.\n\nOne block per paragraph of the entry (marked [P...]), in order: each block verifies numerically what the corresponding statement asserts. Self-contained (numpy/matplotlib only), deterministic.\n\nRequires NumPy and Matplotlib only, and uses fixed seeds, so the printed numbers reproduce exactly. Generated from [`pythondemos/interpretability.py`](https://dictionaryofml.org/terms/interpretability.py); CC BY 4.0."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "# Notebook shim: the script resolves output paths relative to __file__,\n# which a notebook kernel does not define; everything lands in the\n# working directory instead.\nimport os\n__file__ = os.path.join(os.getcwd(), \"interpretability.py\")\nos.makedirs(\"pythondemos\", exist_ok=True)"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "\"\"\"\ninterpretability.py \u2014 numerical companion to the glossary entry\n'interpretability'.\n\nOne block per paragraph of the entry (marked [P...]), in order: each block\nverifies numerically what the corresponding statement asserts.\nSelf-contained (numpy/matplotlib only), deterministic.\n\nSetup\n-----\nh(x)  = 0.4 x + 2                       (a linear map)\nh'(x) = h(x)              for x <= 4    (agrees with h where the user looks)\n      = h(x) - 0.5 (x-4)  for x  > 4    (bends afterwards)\nThe user sees the part of the training set at x in [1, 3], fits a line to it,\nand anticipates the predictions on the test set x in [5.5, 6.5].\n\nBlocks\n------\n[P-cost]     Reading a prediction off the line costs a slope, an intercept\n             and one multiplication; the same prediction from a deep net\n             costs thousands of multiplications, which is the account the\n             user cannot follow.\n[P-judge]    Comprehension is not observable, so interpretability is judged\n             through predictability: the fraction of test points the user\n             anticipates correctly is 1 for h and 0 for h'.\n[P-weaker]   Predictability is weaker than interpretability. A hypothesis\n             built from 200 random pieces agrees with the line to 0.01 on\n             the test set, so the user anticipates its predictions exactly\n             while its computation stays out of reach: anticipating the\n             predictions does not imply being able to carry the computation\n             out.\n[P-figure]   The entry's figure: h is anticipated exactly, h' deviates by at\n             least 0.5 at every test point, and the two agree everywhere in\n             the part of the training set the user has seen.\n[P-decomp]   h decomposes into a slope and an intercept, and the slope states\n             how the prediction changes with the feature -- the difference\n             h(x+1) - h(x) equals it at every x. h' admits no such number: the\n             same difference ranges from -0.10 to 0.40 depending on where it\n             is taken.\n[P-explain]  An explanation can be unfaithful. A line fitted to h' across the\n             bend reports a slope matching neither the part before it nor the\n             part after, so a user who trusts it mispredicts on both sides.\n\nOutputs\n-------\npythondemos/interpretability.png : preview figure (checking only; the entry's\n                                   figure is schematic TikZ).\n\"\"\"\n\nimport numpy as np\nimport matplotlib\n\nmatplotlib.use(\"Agg\")\nimport matplotlib.pyplot as plt\n\nfrom pathlib import Path\n\nOUT_DIR = Path(__file__).parent\n\nrng = np.random.default_rng(20260826)\nreport = []\n\n\ndef check(name, ok):\n    report.append((name, bool(ok)))\n    print(f\"  [{'ok' if ok else 'FAIL'}] {name}\")\n\n\nSLOPE, INTERCEPT, BEND = 0.4, 2.0, 4.0\n\n\ndef h(x):\n    return SLOPE * np.asarray(x, dtype=float) + INTERCEPT\n\n\ndef h_prime(x):\n    x = np.asarray(x, dtype=float)\n    return h(x) - 0.5 * np.maximum(x - BEND, 0.0)\n\n\nx_seen = np.linspace(1.0, 3.0, 6)          # training set the user has seen\nx_test = np.linspace(5.5, 6.5, 5)          # test set to anticipate\n\n\ndef user_line(hyp):\n    \"\"\"The user fits a line to the training set seen, then extrapolates.\"\"\"\n    A = np.c_[x_seen, np.ones_like(x_seen)]\n    w, *_ = np.linalg.lstsq(A, hyp(x_seen), rcond=None)\n    return w\n\n\ndef anticipate(hyp, xs):\n    w = user_line(hyp)\n    return np.c_[xs, np.ones_like(xs)] @ w"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[P-cost]** Reading a prediction off the line costs a slope, an intercept and one multiplication; the same prediction from a deep net costs thousands of multiplications, which is the account the user cannot follow."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[P-cost] what a prediction costs the user to reproduce\")\n\nx0 = 5.9\nby_hand = SLOPE * x0 + INTERCEPT             # one multiplication, one addition\ncheck(\"the line's prediction is one multiplication and one addition\",\n      np.isclose(by_hand, float(h(x0))))\nWIDTHS = (1, 64, 64, 1)                      # a small deep net\nnet_mults = sum(a * b for a, b in zip(WIDTHS, WIDTHS[1:]))\nprint(f\"    multiplications per prediction: line 1, deep net {net_mults}\")\ncheck(\"the deep net needs more than a thousand multiplications\",\n      net_mults > 1000)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[P-judge]** Comprehension is not observable, so interpretability is judged through predictability: the fraction of test points the user anticipates correctly is 1 for h and 0 for h'."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"\\n[P-judge] interpretability is judged through predictability\")\n\nTOL = 0.05\nhit_h = float(np.mean(np.abs(anticipate(h, x_test) - h(x_test)) < TOL))\nhit_hp = float(np.mean(np.abs(anticipate(h_prime, x_test)\n                              - h_prime(x_test)) < TOL))\nprint(f\"    fraction of test points anticipated correctly: h {hit_h:.2f}, \"\n      f\"h' {hit_hp:.2f}\")\ncheck(\"the user anticipates every prediction of h\", hit_h == 1.0)\ncheck(\"the user anticipates no prediction of h'\", hit_hp == 0.0)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[P-weaker]** Predictability is weaker than interpretability. A hypothesis built from 200 random pieces agrees with the line to 0.01 on the test set, so the user anticipates its predictions exactly while its computation stays out of reach: anticipating the predictions does not imply being able to carry the computation out."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "# A hypothesis assembled from many pieces, fitted to agree with the line.\n# The user anticipates its predictions, but could not carry its computation\n# out: predictable does not imply followable.\nprint(\"\\n[P-weaker] predictability is weaker than interpretability\")\n\nNPIECE = 200\na = rng.uniform(-2.0, 2.0, NPIECE)\nb = rng.uniform(-6.0, 6.0, NPIECE)\n\n\ndef pieces(x):\n    x = np.atleast_1d(np.asarray(x, dtype=float))\n    return np.maximum(0.0, np.outer(x, a) + b)\n\n\ngrid = np.linspace(0.0, 8.0, 400)\ncoef, *_ = np.linalg.lstsq(pieces(grid), h(grid), rcond=None)\n\n\ndef g(x):\n    return pieces(x) @ coef\n\n\ndev_g = float(np.max(np.abs(g(x_test) - h(x_test))))\ng_mults = 2 * NPIECE                         # one per piece, one per weight\nprint(f\"    g agrees with the line to {dev_g:.4f} on the test set, \"\n      f\"using {g_mults} multiplications\")\ncheck(\"the user anticipates g's predictions from the line\", dev_g < 0.01)\ncheck(\"g's computation is far beyond one multiplication\", g_mults > 100)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[P-figure]** The entry's figure: h is anticipated exactly, h' deviates by at least 0.5 at every test point, and the two agree everywhere in the part of the training set the user has seen."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"\\n[P-figure] the entry's figure\")\n\ndev_lin = float(np.max(np.abs(anticipate(h, x_test) - h(x_test))))\ncheck(f\"h is anticipated exactly (max deviation {dev_lin:.1e})\",\n      dev_lin < 1e-10)\ndev_kink = float(np.min(np.abs(anticipate(h_prime, x_test)\n                               - h_prime(x_test))))\ncheck(f\"h' deviates by at least 0.5 at every test point \"\n      f\"(min {dev_kink:.2f})\", dev_kink >= 0.5)\ncheck(\"h and h' agree everywhere the user has looked\",\n      float(np.max(np.abs(h(x_seen) - h_prime(x_seen)))) == 0.0)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[P-decomp]** h decomposes into a slope and an intercept, and the slope states how the prediction changes with the feature -- the difference h(x+1) - h(x) equals it at every x. h' admits no such number: the same difference ranges from -0.10 to 0.40 depending on where it is taken."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"\\n[P-decomp] the slope says how the prediction changes with the feature\")\n\nxs_any = np.linspace(0.0, 8.0, 50)\nstep = h(xs_any + 1.0) - h(xs_any)\ncheck(\"h(x+1) - h(x) equals the slope at every x\",\n      float(np.max(np.abs(step - SLOPE))) < 1e-12)\n# h' has no such number: the same difference depends on where it is taken\nstep_p = h_prime(xs_any + 1.0) - h_prime(xs_any)\nspread = float(np.max(step_p) - np.min(step_p))\nprint(f\"    h'(x+1) - h'(x) ranges over {np.min(step_p):.2f} to \"\n      f\"{np.max(step_p):.2f}\")\ncheck(\"h' has no single number playing the role of a slope\", spread > 0.4)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[P-explain]** An explanation can be unfaithful. A line fitted to h' across the bend reports a slope matching neither the part before it nor the part after, so a user who trusts it mispredicts on both sides."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "# A line fitted across the bend of h' -- the shape a local explanation takes.\nprint(\"\\n[P-explain] an explanation can be unfaithful to the hypothesis\")\n\nwin = np.linspace(BEND - 1.0, BEND + 1.0, 40)\nA = np.c_[win, np.ones_like(win)]\nw_expl, *_ = np.linalg.lstsq(A, h_prime(win), rcond=None)\nslope_expl = float(w_expl[0])\nbefore, after = SLOPE, SLOPE - 0.5\nprint(f\"    explanation reports slope {slope_expl:.2f}; the hypothesis has \"\n      f\"{before:.2f} before the bend and {after:.2f} after\")\ncheck(\"the reported slope matches neither side of the bend\",\n      abs(slope_expl - before) > 0.1 and abs(slope_expl - after) > 0.1)\n\n\n# -------------------------------------------------------------- preview\nxs = np.linspace(0.0, 7.0, 400)\nfig, ax = plt.subplots(1, 2, figsize=(9, 3.2))\n\nax[0].plot(xs, h(xs), \"k-\", lw=1.4, label=r\"$\\hat{h}$, a linear map\")\nax[0].plot(xs, h_prime(xs), \"k--\", lw=1.4, label=r\"$\\hat{h}'$, bending at 4\")\nax[0].plot(x_seen, h(x_seen), \"ko\", ms=5, mfc=\"none\",\n           label=\"training set seen\")\nax[0].plot(x_test, anticipate(h_prime, x_test), \"kx\", ms=6,\n           label=\"user anticipation\")\nax[0].plot(x_test, h_prime(x_test), \"k^\", ms=5, mfc=\"none\",\n           label=r\"$\\hat{h}'$ on the test set\")\nax[0].set_xlabel(\"$x$\")\nax[0].set_ylabel(\"$y$\")\nax[0].set_title(\"[P-figure] anticipating the predictions\", fontsize=9)\nax[0].legend(frameon=False, fontsize=6.5, loc=\"upper left\")\n\nax[1].plot(win, h_prime(win), \"k-\", lw=1.4, label=r\"$\\hat{h}'$\")\nax[1].plot(win, A @ w_expl, \"k:\", lw=1.6,\n           label=f\"explanation, slope {slope_expl:.2f}\")\nax[1].axvline(BEND, color=\"0.6\", lw=0.8, ls=\"--\")\nax[1].annotate(\"bend\", xy=(BEND, float(h_prime(BEND))), xytext=(4.05, 3.35),\n               fontsize=7)\nax[1].set_xlabel(\"$x$\")\nax[1].set_ylabel(\"$y$\")\nax[1].set_title(\"[P-explain] a slope matching neither side\", fontsize=9)\nax[1].legend(frameon=False, fontsize=7, loc=\"upper left\")\n\nfig.tight_layout()\nfig.savefig(OUT_DIR / \"interpretability.png\", dpi=110)\n\nn_ok = sum(ok for _, ok in report)\nprint(f\"\\n{n_ok}/{len(report)} checks pass\")\nprint(\"wrote interpretability.png\")\nif n_ok != len(report):\n    raise SystemExit(1)"
  }
 ]
}