B-ent writes when the script runsDictionary of Applied Machine Learning · explainability
Numerical companion to the entry explainability: it recomputes what the entry states and prints one line per check
Quantifies the (subjective) explainability of a trained hypothesis for a simulated user, via the two measures discussed in the entry: the deviation between the user-anticipated and the actual predictions on a test set, and the empirical conditional entropy of the predictions given the user's anticipations. Providing explanations (LIME-style local linear approximations) raises both measures. Self-contained (numpy/matplotlib only), fixed seed.
Run it with python3 pythondemos/explainability.py, from the repository root — it writes its data files under pythondemos/. Requires NumPy and Matplotlib only, and uses fixed seeds, so the printed numbers reproduce exactly. Download explainability.py
One cell per block of the script: the code, and what that code printed when it last ran here
"""
explainability.py — numerical companion to the glossary entry
'explainability'.
Purpose
-------
Quantifies the (subjective) explainability of a trained hypothesis for a
simulated user, via the two measures discussed in the entry: the deviation
between the user-anticipated and the actual predictions on a test set, and
the empirical conditional entropy of the predictions given the user's
anticipations. Providing explanations (LIME-style local linear
approximations) raises both measures. Self-contained (numpy/matplotlib
only), fixed seed.
Setup
-----
Trained hypothesis: Gaussian-kernel ridge regression fit to m = 30 noisy
samples of a nonlinear function on [-3, 3] — opaque to a user who reasons
in terms of linear maps. The user is simulated as ridge-fitting a linear
map to a small set of labeled examples of the hypothesis' predictions
("mental model"), and anticipating predictions on a test set.
Blocks
------
[B-lin] For a LINEAR trained hypothesis, the simulated user anticipates
its test-set predictions almost exactly (mean squared deviation
< 1e-3): a linear hypothesis is highly explainable to this user.
[B-opaque] For the kernel hypothesis, the user's anticipation deviates
strongly (mean squared deviation > 0.1): low explainability.
[B-expl] Explanations close the gap: given a LIME-style local linear
approximation around each test point (fit to perturbations near
the point), the user's anticipation error drops by a factor
of at least 10 compared with [B-opaque].
[B-ent] The empirical conditional entropy H(prediction | anticipation)
(both discretized into 8 bins) is smaller with explanations
than without: anticipations become more informative about the
predictions.
Outputs
-------
explainability_scatter.csv : test-set points with columns yhat (prediction
of the kernel hypothesis), u_no (user
anticipation without explanations), u_expl
(with explanations), for the entry's pgfplots
figure.
explainability.png : matplotlib preview of that figure (checking
only).
"""
import numpy as np
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
report = []
def check(name, ok):
report.append((name, bool(ok)))
print(f" [{'ok' if ok else 'FAIL'}] {name}")
rng = np.random.default_rng(0)
# ------------------------------------------------- trained hypotheses
m = 30
x_tr = np.sort(rng.uniform(-3.0, 3.0, m))
y_tr = np.sin(1.5 * x_tr) + 0.5 * x_tr + 0.1 * rng.standard_normal(m)
SIGMA, ALPHA = 0.6, 1e-2
def gauss_kernel(p, q):
return np.exp(-(p[:, None] - q[None, :]) ** 2 / (2.0 * SIGMA ** 2))
beta = np.linalg.solve(gauss_kernel(x_tr, x_tr) + ALPHA * m * np.eye(m),
y_tr)
def h_kernel(p):
"""Opaque hypothesis: Gaussian-kernel ridge regression."""
return gauss_kernel(np.atleast_1d(p), x_tr) @ beta
w_lin = np.polyfit(x_tr, y_tr, 1)
def h_linear(p):
"""Interpretable hypothesis: linear map (with intercept)."""
return np.polyval(w_lin, p)
def user_anticipation(h, x_examples, x_test):
"""The simulated user ridge-fits a linear map to labeled examples of
the hypothesis' predictions and extrapolates to the test points."""
A = np.c_[x_examples, np.ones_like(x_examples)]
w = np.linalg.solve(A.T @ A + 1e-6 * np.eye(2), A.T @ h(x_examples))
return np.c_[x_test, np.ones_like(x_test)] @ w
x_ex = np.linspace(-3.0, 3.0, 6) # examples shown to the user
x_te = np.linspace(-2.8, 2.8, 40) # test set to anticipate
For a LINEAR trained hypothesis, the simulated user anticipates its test-set predictions almost exactly (mean squared deviation < 1e-3): a linear hypothesis is highly explainable to this user.
dev_lin = float(np.mean(
(user_anticipation(h_linear, x_ex, x_te) - h_linear(x_te)) ** 2))
check(f"[B-lin] linear hypothesis anticipated (msd {dev_lin:.1e})",
dev_lin < 1e-3)
[ok] [B-lin] linear hypothesis anticipated (msd 3.2e-15)
For the kernel hypothesis, the user's anticipation deviates strongly (mean squared deviation > 0.1): low explainability.
u_no = user_anticipation(h_kernel, x_ex, x_te)
yhat = h_kernel(x_te)
dev_no = float(np.mean((u_no - yhat) ** 2))
check(f"[B-opaque] kernel hypothesis not anticipated (msd {dev_no:.2f})",
dev_no > 0.1)
[ok] [B-opaque] kernel hypothesis not anticipated (msd 0.51)
Explanations close the gap: given a LIME-style local linear approximation around each test point (fit to perturbations near the point), the user's anticipation error drops by a factor of at least 10 compared with [B-opaque].
u_expl = np.empty_like(x_te)
for i, xt in enumerate(x_te):
x_loc = xt + 0.25 * rng.standard_normal(20) # LIME-style perturbations
u_expl[i] = user_anticipation(h_kernel, x_loc,
np.array([xt]))[0]
dev_expl = float(np.mean((u_expl - yhat) ** 2))
check(f"[B-expl] explanations shrink the deviation "
f"({dev_no:.2f} -> {dev_expl:.4f})", dev_no > 10.0 * dev_expl)
[ok] [B-expl] explanations shrink the deviation (0.51 -> 0.0013)
The empirical conditional entropy H(prediction | anticipation) (both discretized into 8 bins) is smaller with explanations than without: anticipations become more informative about the predictions.
def cond_entropy(target, given, bins=8):
"""Empirical conditional entropy H(target | given) in bits."""
lo, hi = min(target.min(), given.min()), max(target.max(), given.max())
edges = np.linspace(lo, hi + 1e-9, bins + 1)
t = np.digitize(target, edges) - 1
g = np.digitize(given, edges) - 1
joint = np.zeros((bins, bins))
for ti, gi in zip(t, g):
joint[ti, gi] += 1
joint /= joint.sum()
pg = joint.sum(axis=0)
with np.errstate(divide="ignore", invalid="ignore"):
cond = joint / pg[None, :]
terms = np.where(joint > 0, joint * np.log2(cond), 0.0)
return float(-terms.sum())
H_no = cond_entropy(yhat, u_no)
H_expl = cond_entropy(yhat, u_expl)
check(f"[B-ent] conditional entropy drops ({H_no:.2f} -> {H_expl:.2f} "
f"bits)", H_expl < H_no)
# ---------------------------------------------------------------- CSV
with open("pythondemos/explainability_scatter.csv", "w") as f:
f.write("yhat,u_no,u_expl\n")
for a, b, c in zip(yhat, u_no, u_expl):
f.write(f"{a:.4f},{b:.4f},{c:.4f}\n")
# -------------------------------------------------------------- preview
fig, ax = plt.subplots(figsize=(4.2, 4.2))
lim = [yhat.min() - 0.3, yhat.max() + 0.3]
ax.plot(lim, lim, "k--", lw=0.8)
ax.plot(yhat, u_no, "ks", mfc="none", ms=4, label="without explanations")
ax.plot(yhat, u_expl, "k.", ms=5, label="with explanations")
ax.set_xlabel("prediction $\\hat{h}(x)$")
ax.set_ylabel("user anticipation")
ax.legend(frameon=False, fontsize=8)
ax.set_aspect("equal")
fig.tight_layout()
fig.savefig("pythondemos/explainability.png", dpi=110)
n_ok = sum(ok for _, ok in report)
print(f"\n{n_ok}/{len(report)} checks pass")
print("wrote pythondemos/explainability_scatter.csv, "
"pythondemos/explainability.png")
if n_ok != len(report):
raise SystemExit(1)
[ok] [B-ent] conditional entropy drops (1.22 -> 0.09 bits) 4/4 checks pass wrote pythondemos/explainability_scatter.csv, pythondemos/explainability.png
B-ent writes when the script runs