B-tune writes when the script runsDictionary of Applied Machine Learning · artificial neural network
Numerical companion to the entry artificial neural network: it recomputes what the entry states and prints one line per check
Numerical companion to the glossary entry 'ann' (artificial neural network). Every quantity is computed twice -- once by the network, once by hand -- so each claim of the entry can be checked: the artificial neuron applies an activation function to the weighted sum of its inputs plus an offset term; the five-node network delivers its prediction from the inputs x1, a1, a2; a deep net without skip connections equals the concatenation of its layer-wise feature transformations; adding a skip connection or tuning a single edge weight changes the hypothesis the network represents.
Run it with python3 ann.py, from any directory — it writes its output files into the current directory. Requires NumPy and Matplotlib only, and uses fixed seeds, so the printed numbers reproduce exactly. Download ann.py · Notebook · Open in Colab
One cell per block of the script: the code, and what that code printed when it last ran here
"""The ANN of the entry's figures, computed number by number: a single
artificial neuron, the five-node network, and a deep net evaluated as a
concatenation of layer-wise feature transformations.
Purpose
-------
Numerical companion to the glossary entry 'ann' (artificial neural
network). Every quantity is computed twice -- once by the network,
once by hand -- so each claim of the entry can be checked: the
artificial neuron applies an activation function to the weighted sum of
its inputs plus an offset term; the five-node network delivers its
prediction from the inputs x1, a1, a2; a deep net without skip
connections equals the concatenation of its layer-wise feature
transformations; adding a skip connection or tuning a single edge
weight changes the hypothesis the network represents.
All edge weights and offsets are fixed constants stated in the code
(no randomness), and the activation function is sigma(z) = max(0, z).
Blocks
------
[B-neuron] A single artificial neuron: output a1 = sigma(w1 x1 + w2 x2
+ b), checked against the hand-computed value.
[B-dag] The five-node network: hidden outputs a1, a2 feed the output
node together with x1; the prediction is checked against the
hand-computed weighted sum.
[B-layers] A deep net with two hidden layers of three neurons each:
evaluating it layer by layer equals the concatenation
Phi3(Phi2(Phi1(x))) of the layer-wise feature
transformations, starting from the feature vector
x = (x1, x2, x3).
[B-skip] A skip connection from the first hidden layer straight to
the output changes the prediction.
[B-tune] The edge weights are tunable model parameters: changing one
of them changes the hypothesis, shown as two prediction
surfaces over the inputs (x1, x2) of the five-node network.
Outputs
-------
ann_surface.csv : x1, x2, h -- prediction of the five-node network
ann_surface_tuned.csv : the same after changing the edge weight w5
ann.png : preview (checking only) -- the neuron's output
for two offsets, and the two prediction surfaces
"""
from pathlib import Path
import numpy as np
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
OUT_DIR = Path(__file__).parent
report = [] # collects (check name, pass/fail) pairs
def check(name, ok): # records and prints one verification
report.append((name, bool(ok)))
print(f" [{'ok' if ok else 'FAIL'}] {name}")
def actfun(z):
"""The activation function sigma(z) = max(0, z)."""
return np.maximum(0.0, z)
A single artificial neuron: output a1 = sigma(w1 x1 + w2 x2 + b), checked against the hand-computed value.
w1, w2, b = 0.8, -0.4, 0.3 # edge weights and offset term
x1, x2 = 1.0, 2.0
a1 = actfun(w1 * x1 + w2 * x2 + b)
check("[B-neuron] a1 = sigma(0.8*1.0 - 0.4*2.0 + 0.3) = 0.3",
abs(a1 - 0.3) < 1e-12)
[ok] [B-neuron] a1 = sigma(0.8*1.0 - 0.4*2.0 + 0.3) = 0.3
The five-node network: hidden outputs a1, a2 feed the output node together with x1; the prediction is checked against the hand-computed weighted sum.
W_DAG = dict(w1=0.8, w2=-0.4, w3=0.5, w4=0.9, w5=1.2, w6=-0.7, w7=0.4)
B1, B2 = 0.3, 0.0 # offset terms of the neurons a1, a2
def dag_predict(x1, x2, w):
"""Prediction of the five-node network for the features x1, x2."""
a1 = actfun(w["w1"] * x1 + w["w2"] * x2 + B1)
a2 = actfun(w["w3"] * x1 + w["w4"] * x2 + B2)
return w["w5"] * a1 + w["w6"] * a2 + w["w7"] * x1
h = dag_predict(1.0, 2.0, W_DAG)
check("[B-dag] the prediction is the weighted sum of x1, a1, a2",
abs(h - (1.2 * 0.3 - 0.7 * 2.3 + 0.4 * 1.0)) < 1e-12)
[ok] [B-dag] the prediction is the weighted sum of x1, a1, a2
A deep net with two hidden layers of three neurons each: evaluating it layer by layer equals the concatenation Phi3(Phi2(Phi1(x))) of the layer-wise feature transformations, starting from the feature vector x = (x1, x2, x3).
W1 = np.array([[0.8, -0.4, 0.2], [0.5, 0.9, -0.3], [-0.6, 0.7, 0.4]])
O1 = np.array([0.3, 0.0, 0.1]) # offset terms of the first hidden layer
W2 = np.array([[0.7, -0.5, 0.3], [0.2, 0.6, -0.4], [0.5, 0.1, 0.8]])
O2 = np.array([0.0, 0.2, -0.1]) # offset terms of the second hidden layer
W3 = np.array([1.2, -0.7, 0.4])
O3 = 0.5 # offset term of the output neuron
def phi1(v): # first hidden layer, R^3 -> R^3
return actfun(W1 @ v + O1)
def phi2(v): # second hidden layer, R^3 -> R^3
return actfun(W2 @ v + O2)
def phi3(v): # output layer, R^3 -> R
return float(W3 @ v + O3)
x = np.array([1.0, 2.0, -1.0]) # feature vector x = (x1, x2, x3)
nested = phi3(phi2(phi1(x)))
value = x.copy() # the same net, one layer after the other
for layer in (phi1, phi2):
value = layer(value)
sequential = phi3(value)
check("[B-layers] layer-by-layer evaluation equals Phi3(Phi2(Phi1(x)))",
abs(nested - sequential) < 1e-12)
check("[B-layers] hidden layers map R^3 -> R^3, the output layer to a "
"single prediction",
phi1(x).shape == (3,) and phi2(phi1(x)).shape == (3,)
and np.isscalar(nested))
print(f" Phi1(x) = {phi1(x)}, prediction = {nested:+.3f}")
[ok] [B-layers] layer-by-layer evaluation equals Phi3(Phi2(Phi1(x))) [ok] [B-layers] hidden layers map R^3 -> R^3, the output layer to a single prediction Phi1(x) = [0.1 2.6 0.5], prediction = -0.362
A skip connection from the first hidden layer straight to the output changes the prediction.
W_SKIP = 1.0 # edge weight of the skip connection
h_skip = nested + W_SKIP * phi1(x)[0]
check("[B-skip] adding the skip connection changes the prediction",
abs(h_skip - nested) > 1e-9)
[ok] [B-skip] adding the skip connection changes the prediction
The edge weights are tunable model parameters: changing one of them changes the hypothesis, shown as two prediction surfaces over the inputs (x1, x2) of the five-node network.
W_TUNED = dict(W_DAG, w5=-1.2)
check("[B-tune] changing the edge weight w5 changes the prediction",
abs(dag_predict(1.0, 2.0, W_TUNED) - h) > 1e-9)
g = np.linspace(-2.0, 2.0, 41)
X1, X2 = np.meshgrid(g, g)
H = dag_predict(X1, X2, W_DAG)
H_TUNED = dag_predict(X1, X2, W_TUNED)
for fname, surface in (("ann_surface.csv", H),
("ann_surface_tuned.csv", H_TUNED)):
rows = np.stack([X1.ravel(), X2.ravel(), surface.ravel()], 1)
np.savetxt(OUT_DIR / fname, rows, delimiter=",",
header="x1,x2,h", comments="", fmt="%.4f")
fig, axes = plt.subplots(1, 3, figsize=(11, 3.4))
ax = axes[0]
z1 = np.linspace(-2.0, 2.0, 200)
for offset, style in ((0.3, "-"), (-0.3, "--")):
ax.plot(z1, actfun(w1 * z1 + w2 * x2 + offset), style, color="black",
label=f"offset $b = {offset}$")
ax.set_xlabel("feature $x_1$ (with $x_2 = 2$ fixed)")
ax.set_ylabel("neuron output $a_1$")
ax.set_title("a single artificial neuron")
ax.legend(frameon=False)
for ax, surface, title in ((axes[1], H, "five-node network"),
(axes[2], H_TUNED,
"after tuning edge weight $w_5$")):
filled = ax.contourf(X1, X2, surface, levels=12, cmap="Greys")
lines = ax.contour(X1, X2, surface, levels=6, colors="black",
linewidths=0.6)
ax.clabel(lines, fontsize=6)
ax.set_xlabel("feature $x_1$")
ax.set_ylabel("feature $x_2$")
ax.set_title(f"prediction $h(x_1, x_2)$\n{title}")
fig.tight_layout()
fig.savefig(OUT_DIR / "ann.png", dpi=150)
failed = [name for name, ok in report if not ok]
print(f"{len(report) - len(failed)}/{len(report)} checks passed"
+ (f", FAILED: {failed}" if failed else ""))
[ok] [B-tune] changing the edge weight w5 changes the prediction 6/6 checks passed
B-tune writes when the script runs