B-gd writes when the script runsDictionary of Applied Machine Learning · backpropagation
Numerical companion to the entry backpropagation: it recomputes what the entry states and prints one line per check
Backs the entry's claims on a small artificial neural network (ANN) with one hidden layer, the network of the entry's figure: backpropagation delivers the partial derivative of the loss with respect to every weight in one forward and one backward pass, the result agrees with perturbing each weight separately, and the cost of the latter grows with the number of weights while the cost of backpropagation stays a small multiple of one forward pass. Self-contained (numpy/matplotlib only), fixed seed.
Run it without installing anything:uv run https://dictionaryofml.org/terms/backpropagation.py
uv downloads this script and the pinned NumPy and Matplotlib it needs, then runs it; the script fetches any input file it uses. To keep the output files, download backpropagation.py into a folder and run uv run backpropagation.py there. With NumPy and Matplotlib already installed, python3 backpropagation.py, from any directory — it writes its output files into the current directory. Fixed seeds, so the printed numbers reproduce exactly. Download backpropagation.py · Notebook · Open in Colab
One cell per block of the script: the code, and what that code printed when it last ran here
"""
backpropagation.py — numerical companion to the glossary entry
'backpropagation'.
Purpose
-------
Backs the entry's claims on a small artificial neural network (ANN) with
one hidden layer, the network of the entry's figure: backpropagation
delivers the partial derivative of the loss with respect to every weight
in one forward and one backward pass, the result agrees with perturbing
each weight separately, and the cost of the latter grows with the number
of weights while the cost of backpropagation stays a small multiple of
one forward pass. Self-contained (numpy/matplotlib only), fixed seed.
Setup
-----
Data points with d = 8 features and a numeric label, generated from a
fixed network with added noise. The ANN has one hidden layer of q
neurons with the activation function tanh and a single output neuron;
its weights are collected into one vector w. The loss function is the
squared error, the objective function f(w) the average loss on a batch
of 64 data points. "One pass" counts the multiplications of a forward
pass through the network for the whole batch.
Blocks
------
[B-net] The forward pass: the batch travels through the hidden
layer and the output neuron, and the loss function compares
the predictions to the labels.
[B-backward] The backward pass applies the chain rule from the output
back to the input and yields the whole gradient. Perturbing
each weight by 1e-6 and evaluating f again gives the same
partial derivatives, to within 1e-6.
[B-cost] Multiplications counted: perturbing each weight costs one
forward pass per weight, so the cost grows in proportion to
the number of weights; backpropagation costs between two and
three forward passes whatever the number of weights.
[B-gd] Backpropagation is not itself an iteration: gradient
descent consumes the gradient in a gradient step and calls
backpropagation again. Twenty steps lower f at every step.
Outputs
-------
backpropagation_cost.csv : weights, forward passes needed by perturbation
and by backpropagation, for growing hidden layers
backpropagation_grad.csv : for every weight, the partial derivative from
backpropagation and from perturbation
backpropagation_gd.csv : iteration, objective value along gradient descent
backpropagation.png : matplotlib preview (checking only)
"""
import numpy as np
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
from pathlib import Path
OUT_DIR = Path(__file__).parent
report = []
def check(name, ok):
report.append((name, bool(ok)))
print(f" [{'ok' if ok else 'FAIL'}] {name}")
rng = np.random.default_rng(0)
D, M = 8, 64 # features, batch size
X = rng.standard_normal((M, D))
W_true = rng.standard_normal((D, 6)); v_true = rng.standard_normal(6)
y = np.tanh(X @ W_true) @ v_true + 0.1 * rng.standard_normal(M)
The forward pass: the batch travels through the hidden layer and the output neuron, and the loss function compares the predictions to the labels.
def unpack(w, q):
"""The weight vector w of a network with q hidden neurons, as its layers."""
W1 = w[:D * q].reshape(D, q); b1 = w[D * q:D * q + q]
W2 = w[D * q + q:D * q + 2 * q]; b2 = w[-1]
return W1, b1, W2, b2
def n_weights(q):
return D * q + q + q + 1
def forward(w, q):
"""Forward pass on the batch: hidden activations, predictions, average loss."""
W1, b1, W2, b2 = unpack(w, q)
Z = X @ W1 + b1; H = np.tanh(Z) # hidden layer
pred = H @ W2 + b2 # output neuron
f = float(np.mean((pred - y) ** 2)) # average loss: squared error of the predictions
return H, pred, f
def forward_mults(q):
"""Multiplications of one forward pass for the batch."""
return M * (D * q + q)
Q = 6
w0 = 0.3 * rng.standard_normal(n_weights(Q))
H0, pred0, f0 = forward(w0, Q)
check(f"[B-net] forward pass of a batch of {M} data points through {Q} hidden "
f"neurons: {n_weights(Q)} weights, average loss {f0:.3f}",
pred0.shape == (M,) and np.isfinite(f0))
[ok] [B-net] forward pass of a batch of 64 data points through 6 hidden neurons: 61 weights, average loss 2.315
The backward pass applies the chain rule from the output back to the input and yields the whole gradient. Perturbing each weight by 1e-6 and evaluating f again gives the same partial derivatives, to within 1e-6.
def backward(w, q):
"""Backward pass: the gradient of f, layer by layer from the output back."""
W1, b1, W2, b2 = unpack(w, q)
H, pred, f = forward(w, q)
g_pred = 2.0 * (pred - y) / M # d f / d prediction
gW2 = H.T @ g_pred; gb2 = float(g_pred.sum()) # output layer
gH = np.outer(g_pred, W2) # chain rule into the hidden layer
gZ = gH * (1.0 - H ** 2) # through tanh
gW1 = X.T @ gZ; gb1 = gZ.sum(axis=0)
return np.r_[gW1.ravel(), gb1, gW2, gb2], f
def backward_mults(q):
"""Multiplications of one backward pass (the products above)."""
return M * (q + q + D * q) + M * q
def perturbation_gradient(w, q, eps=1e-6):
"""One extra forward pass per weight."""
base = forward(w, q)[2]; g = np.zeros_like(w)
for k in range(len(w)):
wk = w.copy(); wk[k] += eps
g[k] = (forward(wk, q)[2] - base) / eps
return g
g_bp, _ = backward(w0, Q)
g_pt = perturbation_gradient(w0, Q)
gap = float(np.max(np.abs(g_bp - g_pt)))
check(f"[B-backward] the backward pass yields all {len(w0)} partial derivatives at once, "
f"and they agree with perturbing each weight separately to within {gap:.1e}",
gap < 1e-5)
with open(OUT_DIR / "backpropagation_grad.csv", "w") as fh:
fh.write("weight,backprop,perturbation\n")
for k, (a, b) in enumerate(zip(g_bp, g_pt), 1):
fh.write(f"{k},{a:.6f},{b:.6f}\n")
[ok] [B-backward] the backward pass yields all 61 partial derivatives at once, and they agree with perturbing each weight separately to within 1.0e-06
Multiplications counted: perturbing each weight costs one forward pass per weight, so the cost grows in proportion to the number of weights; backpropagation costs between two and three forward passes whatever the number of weights.
rows = []
for q in (2, 4, 8, 16, 32, 64, 128):
n = n_weights(q); fwd = forward_mults(q)
passes_pt = n + 1 # one forward pass per weight, plus the base
passes_bp = (fwd + backward_mults(q)) / fwd # in units of one forward pass
rows.append((q, n, passes_pt, passes_bp))
check("[B-cost] perturbing each weight costs one forward pass per weight: "
+ ", ".join(f"{n} weights -> {p} passes" for _, n, p, _ in rows[::3]),
all(p == n + 1 for _, n, p, _ in rows))
check("[B-cost] backpropagation costs between two and three forward passes at every "
f"width ({min(r[3] for r in rows):.2f} to {max(r[3] for r in rows):.2f})",
all(2.0 <= r[3] <= 3.0 for r in rows))
with open(OUT_DIR / "backpropagation_cost.csv", "w") as fh:
fh.write("hidden,weights,passes_perturbation,passes_backprop\n")
for q, n, p, b in rows:
fh.write(f"{q},{n},{p},{b:.3f}\n")
[ok] [B-cost] perturbing each weight costs one forward pass per weight: 21 weights -> 22 passes, 161 weights -> 162 passes, 1281 weights -> 1282 passes [ok] [B-cost] backpropagation costs between two and three forward passes at every width (2.22 to 2.22)
Backpropagation is not itself an iteration: gradient descent consumes the gradient in a gradient step and calls backpropagation again. Twenty steps lower f at every step.
LRATE, STEPS = 0.05, 20
w = w0.copy(); fs = [f0]
for t in range(STEPS):
g, _ = backward(w, Q) # backpropagation at the current weights
w = w - LRATE * g # the gradient step consumes it
fs.append(forward(w, Q)[2])
check(f"[B-gd] {STEPS} gradient steps, each calling backpropagation once, lower "
f"the objective at every step: {fs[0]:.3f} -> {fs[-1]:.3f}",
all(b < a for a, b in zip(fs, fs[1:])))
with open(OUT_DIR / "backpropagation_gd.csv", "w") as fh:
fh.write("iteration,objective\n")
for t, v in enumerate(fs):
fh.write(f"{t},{v:.5f}\n")
# -------------------------------------------------------------- preview
fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(12.5, 3.7))
ax1.plot(g_pt, g_bp, "k.", ms=4)
lim = [min(g_pt.min(), g_bp.min()), max(g_pt.max(), g_bp.max())]
ax1.plot(lim, lim, "k--", lw=0.8, label="equal")
ax1.set_xlabel("partial derivative by perturbing one weight")
ax1.set_ylabel("partial derivative by backpropagation")
ax1.set_title(f"the {len(w0)} partial derivatives agree"); ax1.legend(frameon=False)
ax2.plot([r[1] for r in rows], [r[2] for r in rows], "k-o", ms=4, label="perturbing each weight")
ax2.plot([r[1] for r in rows], [r[3] for r in rows], "k--s", ms=4, mfc="none", label="backpropagation")
ax2.set_xscale("log"); ax2.set_yscale("log")
ax2.set_xlabel("number of weights"); ax2.set_ylabel("cost in forward passes")
ax2.set_title("cost of all partial derivatives"); ax2.legend(frameon=False)
ax3.plot(range(STEPS + 1), fs, "k-o", ms=3)
ax3.set_xlabel("gradient step"); ax3.set_ylabel("objective $f(\\mathbf{w})$")
ax3.set_title("gradient descent using backpropagation")
fig.tight_layout()
fig.savefig(OUT_DIR / "backpropagation.png", dpi=110)
n_ok = sum(ok for _, ok in report)
print(f"\n{n_ok}/{len(report)} checks pass")
print(f"wrote {OUT_DIR / 'backpropagation_cost.csv'}, {OUT_DIR / 'backpropagation_grad.csv'}, "
f"{OUT_DIR / 'backpropagation_gd.csv'}, {OUT_DIR / 'backpropagation.png'}")
if n_ok != len(report):
raise SystemExit(1)
[ok] [B-gd] 20 gradient steps, each calling backpropagation once, lower the objective at every step: 2.315 -> 0.785 5/5 checks pass wrote /Users/junga1/dictionaryappliedml/pythondemos/backpropagation_cost.csv, /Users/junga1/dictionaryappliedml/pythondemos/backpropagation_grad.csv, /Users/junga1/dictionaryappliedml/pythondemos/backpropagation_gd.csv, /Users/junga1/dictionaryappliedml/pythondemos/backpropagation.png
B-gd writes when the script runs