P-explain writes when the script runsDictionary of Applied Machine Learning · interpretability
Numerical companion to the entry interpretability: it recomputes what the entry states and prints one line per check
One block per paragraph of the entry (marked [P...]), in order: each block verifies numerically what the corresponding statement asserts. Self-contained (numpy/matplotlib only), deterministic.
Run it without installing anything:uv run https://dictionaryofml.org/terms/interpretability.py
uv downloads this script and the pinned NumPy and Matplotlib it needs, then runs it; the script fetches any input file it uses. To keep the output files, download interpretability.py into a folder and run uv run interpretability.py there. With NumPy and Matplotlib already installed, python3 interpretability.py, from any directory — it writes its output files into the current directory. Fixed seeds, so the printed numbers reproduce exactly. Download interpretability.py · Notebook · Open in Colab
One cell per block of the script: the code, and what that code printed when it last ran here
"""
interpretability.py — numerical companion to the glossary entry
'interpretability'.
One block per paragraph of the entry (marked [P...]), in order: each block
verifies numerically what the corresponding statement asserts.
Self-contained (numpy/matplotlib only), deterministic.
Setup
-----
h(x) = 0.4 x + 2 (a linear map)
h'(x) = h(x) for x <= 4 (agrees with h where the user looks)
= h(x) - 0.5 (x-4) for x > 4 (bends afterwards)
The user sees the part of the training set at x in [1, 3], fits a line to it,
and anticipates the predictions on the test set x in [5.5, 6.5].
Blocks
------
[P-cost] Reading a prediction off the line costs a slope, an intercept
and one multiplication; the same prediction from a deep net
costs thousands of multiplications, which is the account the
user cannot follow.
[P-judge] Comprehension is not observable, so interpretability is judged
through predictability: the fraction of test points the user
anticipates correctly is 1 for h and 0 for h'.
[P-weaker] Predictability is weaker than interpretability. A hypothesis
built from 200 random pieces agrees with the line to 0.01 on
the test set, so the user anticipates its predictions exactly
while its computation stays out of reach: anticipating the
predictions does not imply being able to carry the computation
out.
[P-figure] The entry's figure: h is anticipated exactly, h' deviates by at
least 0.5 at every test point, and the two agree everywhere in
the part of the training set the user has seen.
[P-decomp] h decomposes into a slope and an intercept, and the slope states
how the prediction changes with the feature -- the difference
h(x+1) - h(x) equals it at every x. h' admits no such number: the
same difference ranges from -0.10 to 0.40 depending on where it
is taken.
[P-explain] An explanation can be unfaithful. A line fitted to h' across the
bend reports a slope matching neither the part before it nor the
part after, so a user who trusts it mispredicts on both sides.
Outputs
-------
pythondemos/interpretability.png : preview figure (checking only; the entry's
figure is schematic TikZ).
"""
import numpy as np
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
from pathlib import Path
OUT_DIR = Path(__file__).parent
rng = np.random.default_rng(20260826)
report = []
def check(name, ok):
report.append((name, bool(ok)))
print(f" [{'ok' if ok else 'FAIL'}] {name}")
SLOPE, INTERCEPT, BEND = 0.4, 2.0, 4.0
def h(x):
return SLOPE * np.asarray(x, dtype=float) + INTERCEPT
def h_prime(x):
x = np.asarray(x, dtype=float)
return h(x) - 0.5 * np.maximum(x - BEND, 0.0)
x_seen = np.linspace(1.0, 3.0, 6) # training set the user has seen
x_test = np.linspace(5.5, 6.5, 5) # test set to anticipate
def user_line(hyp):
"""The user fits a line to the training set seen, then extrapolates."""
A = np.c_[x_seen, np.ones_like(x_seen)]
w, *_ = np.linalg.lstsq(A, hyp(x_seen), rcond=None)
return w
def anticipate(hyp, xs):
w = user_line(hyp)
return np.c_[xs, np.ones_like(xs)] @ w
Reading a prediction off the line costs a slope, an intercept and one multiplication; the same prediction from a deep net costs thousands of multiplications, which is the account the user cannot follow.
print("[P-cost] what a prediction costs the user to reproduce")
x0 = 5.9
by_hand = SLOPE * x0 + INTERCEPT # one multiplication, one addition
check("the line's prediction is one multiplication and one addition",
np.isclose(by_hand, float(h(x0))))
WIDTHS = (1, 64, 64, 1) # a small deep net
net_mults = sum(a * b for a, b in zip(WIDTHS, WIDTHS[1:]))
print(f" multiplications per prediction: line 1, deep net {net_mults}")
check("the deep net needs more than a thousand multiplications",
net_mults > 1000)
[P-cost] what a prediction costs the user to reproduce
[ok] the line's prediction is one multiplication and one addition
multiplications per prediction: line 1, deep net 4224
[ok] the deep net needs more than a thousand multiplications
Comprehension is not observable, so interpretability is judged through predictability: the fraction of test points the user anticipates correctly is 1 for h and 0 for h'.
print("\n[P-judge] interpretability is judged through predictability")
TOL = 0.05
hit_h = float(np.mean(np.abs(anticipate(h, x_test) - h(x_test)) < TOL))
hit_hp = float(np.mean(np.abs(anticipate(h_prime, x_test)
- h_prime(x_test)) < TOL))
print(f" fraction of test points anticipated correctly: h {hit_h:.2f}, "
f"h' {hit_hp:.2f}")
check("the user anticipates every prediction of h", hit_h == 1.0)
check("the user anticipates no prediction of h'", hit_hp == 0.0)
[P-judge] interpretability is judged through predictability
fraction of test points anticipated correctly: h 1.00, h' 0.00
[ok] the user anticipates every prediction of h
[ok] the user anticipates no prediction of h'
Predictability is weaker than interpretability. A hypothesis built from 200 random pieces agrees with the line to 0.01 on the test set, so the user anticipates its predictions exactly while its computation stays out of reach: anticipating the predictions does not imply being able to carry the computation out.
# A hypothesis assembled from many pieces, fitted to agree with the line.
# The user anticipates its predictions, but could not carry its computation
# out: predictable does not imply followable.
print("\n[P-weaker] predictability is weaker than interpretability")
NPIECE = 200
a = rng.uniform(-2.0, 2.0, NPIECE)
b = rng.uniform(-6.0, 6.0, NPIECE)
def pieces(x):
x = np.atleast_1d(np.asarray(x, dtype=float))
return np.maximum(0.0, np.outer(x, a) + b)
grid = np.linspace(0.0, 8.0, 400)
coef, *_ = np.linalg.lstsq(pieces(grid), h(grid), rcond=None)
def g(x):
return pieces(x) @ coef
dev_g = float(np.max(np.abs(g(x_test) - h(x_test))))
g_mults = 2 * NPIECE # one per piece, one per weight
print(f" g agrees with the line to {dev_g:.4f} on the test set, "
f"using {g_mults} multiplications")
check("the user anticipates g's predictions from the line", dev_g < 0.01)
check("g's computation is far beyond one multiplication", g_mults > 100)
[P-weaker] predictability is weaker than interpretability
g agrees with the line to 0.0000 on the test set, using 400 multiplications
[ok] the user anticipates g's predictions from the line
[ok] g's computation is far beyond one multiplication
The entry's figure: h is anticipated exactly, h' deviates by at least 0.5 at every test point, and the two agree everywhere in the part of the training set the user has seen.
print("\n[P-figure] the entry's figure")
dev_lin = float(np.max(np.abs(anticipate(h, x_test) - h(x_test))))
check(f"h is anticipated exactly (max deviation {dev_lin:.1e})",
dev_lin < 1e-10)
dev_kink = float(np.min(np.abs(anticipate(h_prime, x_test)
- h_prime(x_test))))
check(f"h' deviates by at least 0.5 at every test point "
f"(min {dev_kink:.2f})", dev_kink >= 0.5)
check("h and h' agree everywhere the user has looked",
float(np.max(np.abs(h(x_seen) - h_prime(x_seen)))) == 0.0)
[P-figure] the entry's figure [ok] h is anticipated exactly (max deviation 8.9e-16) [ok] h' deviates by at least 0.5 at every test point (min 0.75) [ok] h and h' agree everywhere the user has looked
h decomposes into a slope and an intercept, and the slope states how the prediction changes with the feature -- the difference h(x+1) - h(x) equals it at every x. h' admits no such number: the same difference ranges from -0.10 to 0.40 depending on where it is taken.
print("\n[P-decomp] the slope says how the prediction changes with the feature")
xs_any = np.linspace(0.0, 8.0, 50)
step = h(xs_any + 1.0) - h(xs_any)
check("h(x+1) - h(x) equals the slope at every x",
float(np.max(np.abs(step - SLOPE))) < 1e-12)
# h' has no such number: the same difference depends on where it is taken
step_p = h_prime(xs_any + 1.0) - h_prime(xs_any)
spread = float(np.max(step_p) - np.min(step_p))
print(f" h'(x+1) - h'(x) ranges over {np.min(step_p):.2f} to "
f"{np.max(step_p):.2f}")
check("h' has no single number playing the role of a slope", spread > 0.4)
[P-decomp] the slope says how the prediction changes with the feature
[ok] h(x+1) - h(x) equals the slope at every x
h'(x+1) - h'(x) ranges over -0.10 to 0.40
[ok] h' has no single number playing the role of a slope
An explanation can be unfaithful. A line fitted to h' across the bend reports a slope matching neither the part before it nor the part after, so a user who trusts it mispredicts on both sides.
# A line fitted across the bend of h' -- the shape a local explanation takes.
print("\n[P-explain] an explanation can be unfaithful to the hypothesis")
win = np.linspace(BEND - 1.0, BEND + 1.0, 40)
A = np.c_[win, np.ones_like(win)]
w_expl, *_ = np.linalg.lstsq(A, h_prime(win), rcond=None)
slope_expl = float(w_expl[0])
before, after = SLOPE, SLOPE - 0.5
print(f" explanation reports slope {slope_expl:.2f}; the hypothesis has "
f"{before:.2f} before the bend and {after:.2f} after")
check("the reported slope matches neither side of the bend",
abs(slope_expl - before) > 0.1 and abs(slope_expl - after) > 0.1)
# -------------------------------------------------------------- preview
xs = np.linspace(0.0, 7.0, 400)
fig, ax = plt.subplots(1, 2, figsize=(9, 3.2))
ax[0].plot(xs, h(xs), "k-", lw=1.4, label=r"$\hat{h}$, a linear map")
ax[0].plot(xs, h_prime(xs), "k--", lw=1.4, label=r"$\hat{h}'$, bending at 4")
ax[0].plot(x_seen, h(x_seen), "ko", ms=5, mfc="none",
label="training set seen")
ax[0].plot(x_test, anticipate(h_prime, x_test), "kx", ms=6,
label="user anticipation")
ax[0].plot(x_test, h_prime(x_test), "k^", ms=5, mfc="none",
label=r"$\hat{h}'$ on the test set")
ax[0].set_xlabel("$x$")
ax[0].set_ylabel("$y$")
ax[0].set_title("[P-figure] anticipating the predictions", fontsize=9)
ax[0].legend(frameon=False, fontsize=6.5, loc="upper left")
ax[1].plot(win, h_prime(win), "k-", lw=1.4, label=r"$\hat{h}'$")
ax[1].plot(win, A @ w_expl, "k:", lw=1.6,
label=f"explanation, slope {slope_expl:.2f}")
ax[1].axvline(BEND, color="0.6", lw=0.8, ls="--")
ax[1].annotate("bend", xy=(BEND, float(h_prime(BEND))), xytext=(4.05, 3.35),
fontsize=7)
ax[1].set_xlabel("$x$")
ax[1].set_ylabel("$y$")
ax[1].set_title("[P-explain] a slope matching neither side", fontsize=9)
ax[1].legend(frameon=False, fontsize=7, loc="upper left")
fig.tight_layout()
fig.savefig(OUT_DIR / "interpretability.png", dpi=110)
n_ok = sum(ok for _, ok in report)
print(f"\n{n_ok}/{len(report)} checks pass")
print("wrote interpretability.png")
if n_ok != len(report):
raise SystemExit(1)
[P-explain] an explanation can be unfaithful to the hypothesis
explanation reports slope 0.15; the hypothesis has 0.40 before the bend and -0.10 after
[ok] the reported slope matches neither side of the bend
12/12 checks pass
wrote interpretability.png
P-explain writes when the script runs