B-plot writes when the script runsDictionary of Applied Machine Learning · pretraining
Updated on 2026-11-03
Numerical companion to the entry pretraining: it recomputes what the entry states and prints one line per check
Numerical companion to the glossary entry 'pretraining'. A 2.4 x 1.2 km piece of the aerial photograph of the Wachau (assets/ wachau_ortho_center.jpg: the Danube bend at Duernstein, 0.79 m per pixel; orthophoto (c) basemap.at, CC BY 4.0) is cut into 4608 square patches of 32 px (25 m), and the parcel mask assets/ wachau_labels_center.png (INVEKOS Schlaege 2025-1, AgrarMarkt Austria, CC BY 3.0 AT) labels a patch as vineyard when more than half of its pixels lie on vineyard parcels. The western half of the piece is the training set, the eastern half the test set. At this resolution the rows of vines are visible stripes, and the mean color of a patch tells little: vineyards, meadows and gardens are all green.
Run it without installing anything:uv run https://dictionaryofml.org/terms/pretraining.py
uv downloads this script and the pinned NumPy and Matplotlib it needs, then runs it; the script fetches any input file it uses. To keep the output files, download pretraining.py into a folder and run uv run pretraining.py there. With NumPy and Matplotlib already installed, python3 pretraining.py, from any directory — it writes its output files into the current directory. Fixed seeds, so the printed numbers reproduce exactly. Download pretraining.py · Notebook · Open in Colab
Before running a downloaded copy, scan it: uvx bandit -q -t B102,B307,B602,B605 pretraining.py (B102,B307,B602,B605 are the exec, eval and shell checks of bandit) reports nothing: the script runs no shell command and evaluates no generated code.
One cell per block of the script: the code, and what that code printed when it last ran here
"""Pretraining the first layers of a convolutional network on a pretext
task without labels, then reusing them as frozen features for a linear
classifier of vineyard patches.
Purpose
-------
Numerical companion to the glossary entry 'pretraining'. A 2.4 x 1.2 km
piece of the aerial photograph of the Wachau (assets/
wachau_ortho_center.jpg: the Danube bend at Duernstein, 0.79 m per
pixel; orthophoto (c) basemap.at, CC BY 4.0) is cut into 4608 square
patches of 32 px (25 m), and the parcel mask assets/
wachau_labels_center.png (INVEKOS Schlaege 2025-1, AgrarMarkt Austria,
CC BY 3.0 AT) labels a patch as vineyard when more than half of its
pixels lie on vineyard parcels. The western half of the piece is the
training set, the eastern half the test set. At this resolution the
rows of vines are visible stripes, and the mean color of a patch tells
little: vineyards, meadows and gardens are all green.
Pretraining needs no labels. Two pretext tasks are run on the training
patches, each with the same network: two convolutional layers (3 x 3
filters, 8 channels each, ReLU) and a linear readout. The first blanks
the central 8 x 8 pixel block of a patch and predicts the mean color of
the blanked pixels from their surroundings. The second rotates a patch
by 0, 90, 180 or 270 degrees and predicts the rotation, which is
answered by the direction of the rows and of the shadows. Only the
photograph is used, never the parcel mask.
Transfer. The two convolutional layers are then frozen and applied to
the original patches; their output, average-pooled to a 4 x 4 grid,
gives 128 features per patch. A linear classifier (logistic regression
with a small ridge penalty) is trained on these features of as few as 8
and at most 128 labeled western patches and evaluated on the eastern
patches. Labels are scarce on purpose: that is the regime in which
pretraining decides the outcome, and in which a classifier on the pixels
themselves, which has more features than labeled patches, overfits to no
better than the majority baseline. The
comparison: the same linear classifier on the mean color of the patch
(3 features), on the pixels pooled to 16 x 16 (768 features), and on the
features of
the same two layers with their random initial weights, i.e., without
pretraining; and the majority baseline. The last comparison is the
control: the color-pretrained layers are ahead of the untrained ones from
32 labels on, and the rotation-pretrained layers beat them at every
number of labels.
Deterministic: fixed seeds for the initial weights and the patches of each step.
Self-contained: numpy + matplotlib only.
Blocks
------
[B-patches] Cut the photograph into 4608 patches of 32 px (25 m), label
them by the parcel mask, and split them by easting into a
western training half and an eastern test half.
[B-pretrain] Blank the center of every training patch and train the two
convolutional layers plus a linear readout to predict the
center color (Adam, 512 patches per step). Check that the
loss falls and that the trained network beats the patch's
mean color as a prediction of the center on the test half.
[B-rotation] The second pretext task: predict the rotation of a patch
(Adam, 512 patches per step). Check that the rotation is
recognized well above chance on the test half.
[B-transfer] Freeze the pretrained layers, pool their output to 128
features, and fit a linear classifier of vineyard vs. no
vineyard on 8 to 128 labeled patches of the training half; compare its test accuracy with the same classifier on
mean color, on raw pixels and on the features of the
untrained layers, and with the majority baseline.
[B-plot] Write the CSV files and images for the entry's figures and
the preview, among them the photograph beside its parcel
register.
Outputs
-------
pretraining_loss.csv : iteration, train, test -- the color pretext loss
(mean squared error of the predicted center
color, in units of the pixel range [0, 1]) on the
two halves
pretraining_accuracy.csv : labels, meancolor, rawpixels, untrained,
color, rotation, baseline -- the test accuracy of
the linear classifier on each feature set against
the number of labeled patches (mean over up to
five spread-out subsets), and the majority
baseline
pretraining_scene.png : the piece of the photograph and the parcel
register beside it, with the western training
half and the eastern test half marked
pretraining_patches.png : eight test patches: the blanked input, the
network's prediction of the center painted in,
and the original
pretraining.png : preview (checking only)
"""
# /// script
# requires-python = ">=3.10"
# dependencies = ["matplotlib==3.10.7", "numpy==2.3.4"]
# ///
import math
from pathlib import Path
import urllib.request
import numpy as np
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
from matplotlib.image import imread
OUT_DIR = Path(__file__).parent
ASSET_URL = "https://dictionaryofml.org/demoassets/"
def demo_asset(name):
"""Path to the input file `name`, fetched once if it is not already here.
Inside the repository the committed copy is used (pythondemos/ first,
then assets/), so a run there reproduces the committed outputs exactly.
Run from anywhere else -- from the copy published on dictionaryofml.org,
say -- the file is downloaded and cached beside the script, so the demo
is runnable without the repository.
"""
for cand in (OUT_DIR / name, OUT_DIR.parent / "assets" / name):
if cand.exists():
return cand
cached = OUT_DIR / name
print(f" fetching {name} from {ASSET_URL}{name}")
with urllib.request.urlopen(ASSET_URL + name, timeout=180) as resp:
cached.write_bytes(resp.read())
return cached
report = [] # collects (check name, pass/fail) pairs
def check(name, ok): # records and prints one verification
report.append((name, bool(ok)))
print(f" [{'ok' if ok else 'FAIL'}] {name}")
4608 patches of 48 x 96, 32 x 32 px each; vineyard share 0.326 (west 0.343, east 0.308)
Cut the photograph into 4608 patches of 32 px (25 m), label them by the parcel mask, and split them by easting into a western training half and an eastern test half.
SIDE = 32 # 32 px = 25 m on the ground
photo = imread(demo_asset("wachau_ortho_center.jpg")) / 255.0
labels = np.asarray(imread(demo_asset("wachau_labels_center.png")))
if labels.ndim == 3:
labels = labels[:, :, 0]
if labels.max() <= 1.0 and labels.dtype != np.uint8:
labels = np.rint(labels * 255).astype(np.uint8)
rows, cols = photo.shape[0] // SIDE, photo.shape[1] // SIDE
patches = photo[:rows * SIDE, :cols * SIDE].reshape(rows, SIDE, cols, SIDE, 3)
patches = patches.transpose(0, 2, 1, 3, 4).reshape(rows * cols, SIDE, SIDE, 3).astype(np.float32)
share = (labels[:rows * SIDE, :cols * SIDE] == 1).reshape(
rows, SIDE, cols, SIDE).mean(axis=(1, 3)).ravel()
y = (share > 0.5).astype(np.float32) # 1 = vineyard
grid_c = np.arange(rows * cols) % cols
train = grid_c < cols // 2 # western half
test = ~train
mean_color = patches[train].mean(axis=(0, 1, 2)) # training statistics only
X_img = patches - mean_color # centered pixels
print(f" {len(y)} patches of {rows} x {cols}, {SIDE} x {SIDE} px each; vineyard share "
f"{y.mean():.3f} (west {y[train].mean():.3f}, east {y[test].mean():.3f})")
check("[B-patches] 4608 patches, split into halves of equal size by easting",
len(y) == 4608 and train.sum() == test.sum() == 2304)
check("[B-patches] between a fifth and two fifths of the patches are vineyard",
0.2 < y.mean() < 0.4)
[ok] [B-patches] 4608 patches, split into halves of equal size by easting [ok] [B-patches] between a fifth and two fifths of the patches are vineyard iteration 1: color pretext loss train 0.00621, test 0.01020 iteration 50: color pretext loss train 0.00058, test 0.00105 iteration 100: color pretext loss train 0.00046, test 0.00086 iteration 150: color pretext loss train 0.00040, test 0.00078 iteration 200: color pretext loss train 0.00036, test 0.00074 iteration 250: color pretext loss train 0.00032, test 0.00072 iteration 300: color pretext loss train 0.00030, test 0.00076 center color on the test half: mean squared error of the network 0.00076, of the patch's mean color outside the blank 0.00296
Blank the center of every training patch and train the two convolutional layers plus a linear readout to predict the center color (Adam, 512 patches per step). Check that the loss falls and that the trained network beats the patch's mean color as a prediction of the center on the test half.
rng = np.random.default_rng(0)
CH = 8 # channels of both convolutional layers
C0, C1 = 12, 20 # the blanked center block: rows and columns 12..19
ITERS, BATCH, LR, B1, B2 = 300, 512, 3e-3, 0.9, 0.999
def conv_cols(x):
"""3 x 3 neighborhoods of every pixel (zero padding): (N, H, W, 9 C)."""
n, h, w, c = x.shape
xp = np.pad(x, ((0, 0), (1, 1), (1, 1), (0, 0)))
return np.concatenate([xp[:, i:i + h, j:j + w, :] for i in range(3) for j in range(3)],
axis=3)
def conv_cols_back(dcols, shape):
"""Adjoint of conv_cols: scatter the neighborhood gradients back to pixels."""
n, h, w, c = shape
dxp = np.zeros((n, h + 2, w + 2, c), dtype=np.float32)
for k, (i, j) in enumerate((i, j) for i in range(3) for j in range(3)):
dxp[:, i:i + h, j:j + w, :] += dcols[..., k * c:(k + 1) * c]
return dxp[:, 1:-1, 1:-1, :]
def init_params(n_in, n_out):
"""He initialization of the two convolutional layers and a linear readout."""
w1 = (rng.standard_normal((27, CH)) * math.sqrt(2.0 / 27)).astype(np.float32)
w2 = (rng.standard_normal((9 * CH, CH)) * math.sqrt(2.0 / (9 * CH))).astype(np.float32)
wh = (rng.standard_normal((n_in * CH, n_out)) * math.sqrt(1.0 / (n_in * CH))).astype(np.float32)
return {"w1": w1, "b1": np.zeros(CH, np.float32), "w2": w2,
"b2": np.zeros(CH, np.float32), "wh": wh, "bh": np.zeros(n_out, np.float32)}
def layers(p, x):
"""The two convolutional layers with ReLU: their output (N, 32, 32, CH)."""
c1 = conv_cols(x)
a1 = np.maximum(c1 @ p["w1"] + p["b1"], 0.0)
c2 = conv_cols(a1)
a2 = np.maximum(c2 @ p["w2"] + p["b2"], 0.0)
return a2, (c1, a1, c2)
def layers_back(p, da2, saved, grads):
"""Backpropagate a gradient at the layers' output into their weights."""
c1, a1, c2 = saved
grads["w2"] = c2.reshape(-1, 9 * CH).T @ da2.reshape(-1, CH)
grads["b2"] = da2.sum((0, 1, 2))
da1 = conv_cols_back(da2 @ p["w2"].T, a1.shape)
da1 *= a1 > 0
grads["w1"] = c1.reshape(-1, 27).T @ da1.reshape(-1, CH)
grads["b1"] = da1.sum((0, 1, 2))
return grads
def adam_step(p, grads, moments, step):
for k in p:
m, v = moments[k]
m[...] = B1 * m + (1 - B1) * grads[k]
v[...] = B2 * v + (1 - B2) * grads[k] ** 2
p[k] -= LR * (m / (1 - B1 ** step)) / (np.sqrt(v / (1 - B2 ** step)) + 1e-8)
def color_forward(p, x):
a2, saved = layers(p, x)
block = a2[:, C0:C1, C0:C1, :].reshape(len(x), -1)
return block @ p["wh"] + p["bh"], (a2, block, saved)
def color_backward(p, target, out, cache):
"""Gradients of the mean squared error of the center color."""
a2, block, saved = cache
n = len(out)
dout = 2.0 * (out - target) / (n * 3)
grads = {"wh": block.T @ dout, "bh": dout.sum(0)}
da2 = np.zeros_like(a2)
da2[:, C0:C1, C0:C1, :] = (dout @ p["wh"].T).reshape(n, C1 - C0, C1 - C0, CH)
da2 *= a2 > 0
return layers_back(p, da2, saved, grads)
def blank(x):
"""The pretext input: the center block set to zero (the mean color)."""
xb = x.copy()
xb[:, C0:C1, C0:C1, :] = 0.0
return xb
def in_slices(fn, x, size=256):
"""Apply fn to x in slices, so the 9-fold neighborhood expansion inside
the layers is held for a few hundred patches at a time rather than for
all of them."""
return np.concatenate([fn(x[i:i + size]) for i in range(0, len(x), size)])
X_blank = blank(X_img)
target = X_img[:, C0:C1, C0:C1, :].mean(axis=(1, 2)) # the color to predict
params_color = init_params((C1 - C0) ** 2, 3)
params_random = {k: v.copy() for k, v in params_color.items()} # the untrained control
moments = {k: (np.zeros_like(v), np.zeros_like(v)) for k, v in params_color.items()}
train_idx = np.flatnonzero(train)
loss_log = []
for step in range(1, ITERS + 1):
sel = np.sort(rng.choice(train_idx, BATCH, replace=False))
out, cache = color_forward(params_color, X_blank[sel])
adam_step(params_color, color_backward(params_color, target[sel], out, cache), moments, step)
if step == 1 or step % 50 == 0:
err = [float(((in_slices(lambda z: color_forward(params_color, z)[0], X_blank[part])
- target[part]) ** 2).mean()) for part in (train, test)]
loss_log.append((step, err[0], err[1]))
print(f" iteration {step:3d}: color pretext loss train {err[0]:.5f}, "
f"test {err[1]:.5f}")
mse_net = loss_log[-1][2]
ring_mean = X_blank[test].sum(axis=(1, 2)) / (SIDE * SIDE - (C1 - C0) ** 2)
mse_ring = float(((ring_mean - target[test]) ** 2).mean())
print(f" center color on the test half: mean squared error of the network {mse_net:.5f}, "
f"of the patch's mean color outside the blank {mse_ring:.5f}")
check("[B-pretrain] the color pretext loss on the training half falls to below a "
"tenth of its initial value", loss_log[-1][1] < 0.1 * loss_log[0][1])
check("[B-pretrain] on the test half the network predicts the center color better "
"than the patch's mean color outside the blank", mse_net < mse_ring)
[ok] [B-pretrain] the color pretext loss on the training half falls to below a tenth of its initial value [ok] [B-pretrain] on the test half the network predicts the center color better than the patch's mean color outside the blank iteration 1: rotation recognized in 0.281 of the step's patches iteration 100: rotation recognized in 0.357 of the step's patches iteration 200: rotation recognized in 0.449 of the step's patches iteration 300: rotation recognized in 0.459 of the step's patches rotation recognized in 0.368 of the rotated test patches (chance 0.25)
The second pretext task: predict the rotation of a patch (Adam, 512 patches per step). Check that the rotation is recognized well above chance on the test half.
def pool4(a2):
"""Average-pool the layers' output to a 4 x 4 grid: 128 numbers per patch."""
n = len(a2)
return a2.reshape(n, 4, SIDE // 4, 4, SIDE // 4, CH).mean(axis=(2, 4)).reshape(n, -1)
def rotation_forward(p, x):
a2, saved = layers(p, x)
pooled = pool4(a2)
return pooled @ p["wh"] + p["bh"], (a2, pooled, saved)
def rotation_backward(p, target, logits, cache):
"""Gradients of the cross-entropy of the four rotations."""
a2, pooled, saved = cache
n = len(logits)
q = np.exp(logits - logits.max(1, keepdims=True))
q /= q.sum(1, keepdims=True)
dout = (q - np.eye(4, dtype=np.float32)[target]) / n
grads = {"wh": pooled.T @ dout, "bh": dout.sum(0)}
dpool = (dout @ p["wh"].T).reshape(n, 4, 1, 4, 1, CH) / (SIDE // 4) ** 2
da2 = np.broadcast_to(dpool, (n, 4, SIDE // 4, 4, SIDE // 4, CH)).reshape(a2.shape).copy()
da2 *= a2 > 0
return layers_back(p, da2, saved, grads)
def rotations(x):
"""Every patch in the four rotations, with the rotation as label."""
return (np.concatenate([np.rot90(x, k, axes=(1, 2)) for k in range(4)]),
np.repeat(np.arange(4), len(x)))
X_rot, rot_label = rotations(X_img[train])
params_rot = init_params(16, 4)
moments = {k: (np.zeros_like(v), np.zeros_like(v)) for k, v in params_rot.items()}
for step in range(1, ITERS + 1):
sel = np.sort(rng.choice(len(X_rot), BATCH, replace=False))
logits, cache = rotation_forward(params_rot, X_rot[sel])
adam_step(params_rot, rotation_backward(params_rot, rot_label[sel], logits, cache),
moments, step)
if step == 1 or step % 100 == 0:
print(f" iteration {step:3d}: rotation recognized in "
f"{(logits.argmax(1) == rot_label[sel]).mean():.3f} of the step's patches")
X_rot_test, rot_label_test = rotations(X_img[test])
rot_acc = float((in_slices(lambda z: rotation_forward(params_rot, z)[0], X_rot_test).argmax(1)
== rot_label_test).mean())
print(f" rotation recognized in {rot_acc:.3f} of the rotated test patches (chance 0.25)")
check("[B-rotation] the rotation of a test patch is recognized in at least 35% of "
"the cases, against 25% by chance", rot_acc >= 0.35)
[ok] [B-rotation] the rotation of a test patch is recognized in at least 35% of the cases, against 25% by chance
8 labeled patches: test accuracy mean color 0.721, raw pixels 0.642, untrained layers 0.717, color-pretrained layers 0.726, rotation-pretrained layers 0.765
16 labeled patches: test accuracy mean color 0.702, raw pixels 0.656, untrained layers 0.738, color-pretrained layers 0.742, rotation-pretrained layers 0.781
32 labeled patches: test accuracy mean color 0.676, raw pixels 0.668, untrained layers 0.729, color-pretrained layers 0.756, rotation-pretrained layers 0.787
64 labeled patches: test accuracy mean color 0.661, raw pixels 0.676, untrained layers 0.743, color-pretrained layers 0.764, rotation-pretrained layers 0.812
128 labeled patches: test accuracy mean color 0.668, raw pixels 0.719, untrained layers 0.764, color-pretrained layers 0.784, rotation-pretrained layers 0.818
majority baseline (always 'no vineyard'): test accuracy 0.692
Freeze the pretrained layers, pool their output to 128 features, and fit a linear classifier of vineyard vs. no vineyard on 8 to 128 labeled patches of the training half; compare its test accuracy with the same classifier on mean color, on raw pixels and on the features of the untrained layers, and with the majority baseline.
def logistic_regression(x_tr, y_tr, ridge=1e-2, iters=30):
"""Logistic regression with a ridge penalty on standardized features.
With fewer labeled patches than features the Newton step is taken in the
span of the patches instead of in feature space. Writing the scaled
design as U S V^T, the weights that minimize the penalized loss are V a
for some a, and the penalty is |a|^2 because V has orthonormal columns,
so fitting on U S is the same problem in as many unknowns as there are
patches: a 129 x 129 solve rather than a 769 x 769 one."""
mu, sd = x_tr.mean(0), x_tr.std(0) + 1e-6 # training statistics
scaled = ((x_tr - mu) / sd).astype(np.float64)
basis = None
if scaled.shape[1] > len(scaled):
left, sing, right = np.linalg.svd(scaled, full_matrices=False)
scaled, basis = left * sing, right
z = np.column_stack([np.ones(len(scaled)), scaled])
w = np.zeros(z.shape[1])
reg = ridge * np.eye(z.shape[1])
reg[0, 0] = 0.0
for _ in range(iters):
q = 1.0 / (1.0 + np.exp(-z @ w))
grad = z.T @ (q - y_tr) / len(z) + reg @ w
hess = (z * (q * (1 - q))[:, None]).T @ z / len(z) + reg
w -= np.linalg.solve(hess, grad)
offset, weights = w[0], (w[1:] @ basis if basis is not None else w[1:])
return lambda x: (offset + (x - mu) / sd @ weights > 0).astype(np.float32)
def frozen_features(p):
return in_slices(lambda z: pool4(layers(p, z)[0]), X_img)
# the pixels themselves, pooled 2 x 2 to 16 x 16: a linear classifier on all
# 3072 pixels needs a 3073 x 3073 Newton solve per step, minutes per fit, and
# scores the same as this cheaper version of the same baseline
feature_sets = {
"mean color": patches.mean(axis=(1, 2)),
"raw pixels": X_img.reshape(len(y), 16, 2, 16, 2, 3).mean(axis=(2, 4)).reshape(len(y), -1),
"untrained layers": frozen_features(params_random),
"color-pretrained layers": frozen_features(params_color),
"rotation-pretrained layers": frozen_features(params_rot),
}
majority = 1.0 if y[train].mean() >= 0.5 else 0.0 # the more frequent label
acc_base = float((majority == y[test]).mean())
# Few labels on purpose. With a hundred labeled patches or fewer the
# classifier cannot learn what the pretext task already taught the layers,
# which is the regime where pretraining decides the outcome; with a
# thousand it no longer matters what the layers were initialized with.
SIZES = (8, 16, 32, 64, 128) # labeled patches used
acc = {name: [] for name in feature_sets} # test accuracy against the size
for m in SIZES:
stride = len(train_idx) // m
for name, f in feature_sets.items():
runs = []
for offset in range(min(10, stride)): # up to ten spread-out subsets
idx = train_idx[offset::stride][:m]
predict = logistic_regression(f[idx], y[idx])
runs.append(float((predict(f[test]) == y[test]).mean()))
acc[name].append(float(np.mean(runs)))
print(f" {m:4d} labeled patches: test accuracy "
+ ", ".join(f"{name} {acc[name][-1]:.3f}" for name in feature_sets))
print(f" majority baseline (always 'no vineyard'): test accuracy {acc_base:.3f}")
check("[B-transfer] a linear classifier on the raw pixels, which has more "
"features than labeled patches, is no better than answering with the "
"majority at the smallest label counts",
all(a <= acc_base for a in acc["raw pixels"][:3]))
check("[B-transfer] the rotation-pretrained layers beat the untrained ones at every "
"number of labeled patches, and are never below the color-pretrained ones",
all(r > u and r >= c for r, u, c in zip(acc["rotation-pretrained layers"],
acc["untrained layers"],
acc["color-pretrained layers"])))
check("[B-transfer] from sixteen labeled patches on, both pretrained layer "
"sets beat mean color and raw pixels",
all(min(c, r) > max(a, b) for c, r, a, b in
zip(acc["color-pretrained layers"][1:], acc["rotation-pretrained layers"][1:],
acc["mean color"][1:], acc["raw pixels"][1:])))
check("[B-transfer] the mean color of a patch stays within 0.05 of the majority "
"baseline at every number of labeled patches: color alone does not tell a "
"vineyard", all(abs(a - acc_base) <= 0.05 for a in acc["mean color"]))
[ok] [B-transfer] a linear classifier on the raw pixels, which has more features than labeled patches, is no better than answering with the majority at the smallest label counts [ok] [B-transfer] the rotation-pretrained layers beat the untrained ones at every number of labeled patches, and are never below the color-pretrained ones [ok] [B-transfer] from sixteen labeled patches on, both pretrained layer sets beat mean color and raw pixels [ok] [B-transfer] the mean color of a patch stays within 0.05 of the majority baseline at every number of labeled patches: color alone does not tell a vineyard
Write the CSV files and images for the entry's figures and the preview, among them the photograph beside its parcel register.
with open(OUT_DIR / "pretraining_loss.csv", "w") as f:
f.write("iteration,train,test\n")
for step, tr, te in loss_log:
f.write(f"{step},{tr:.6f},{te:.6f}\n")
with open(OUT_DIR / "pretraining_accuracy.csv", "w") as f:
f.write("labels,meancolor,rawpixels,untrained,color,rotation,baseline\n")
for i, m in enumerate(SIZES):
f.write(f"{m},{acc['mean color'][i]:.4f},{acc['raw pixels'][i]:.4f},"
f"{acc['untrained layers'][i]:.4f},{acc['color-pretrained layers'][i]:.4f},"
f"{acc['rotation-pretrained layers'][i]:.4f},{acc_base:.4f}\n")
KM_X, KM_Y = cols * SIDE * 0.79 / 1000, rows * SIDE * 0.79 / 1000
split = KM_X / 2 # the line between the halves
vineyard = (labels[:rows * SIDE, :cols * SIDE] == 1)
fig, axes = plt.subplots(1, 2, figsize=(8, 2.6))
for ax, img, cmap, title in (
(axes[0], photo[:rows * SIDE, :cols * SIDE], None,
"Aerial photograph, 0.79 m per pixel"),
(axes[1], vineyard, "gray", "Parcel register: vineyard parcels (white)")):
ax.imshow(img, cmap=cmap, extent=[0, KM_X, 0, KM_Y])
ax.axvline(split, color="black", linestyle="--", linewidth=1.2)
ax.set_title(title, fontsize=11)
ax.set_xlabel("easting (km)", fontsize=10)
ax.set_ylabel("northing (km)", fontsize=10)
ax.tick_params(labelsize=8)
ax.text(split / 2, 0.06 * KM_Y, "western half:\ntraining set", fontsize=9,
ha="center", va="bottom", color="black",
bbox=dict(facecolor="white", edgecolor="none", pad=1.5))
ax.text(1.5 * split, 0.06 * KM_Y, "eastern half:\ntest set", fontsize=9,
ha="center", va="bottom", color="black",
bbox=dict(facecolor="white", edgecolor="none", pad=1.5))
fig.tight_layout()
fig.savefig(OUT_DIR / "pretraining_scene.png", dpi=150)
plt.close(fig)
show = np.flatnonzero(test)[::288][:8] # eight test patches, spread out
pred_show, _ = color_forward(params_color, X_blank[show])
GAP = 3
strip = np.ones((3 * SIDE + 2 * GAP, 8 * SIDE + 7 * GAP, 3), np.float32)
for i, idx in enumerate(show):
col = i * (SIDE + GAP)
blanked = X_blank[idx] + mean_color
blanked[C0:C1, C0:C1, :] = 1.0 # the blank shown white
filled = blanked.copy()
filled[C0:C1, C0:C1, :] = pred_show[i] + mean_color
for r, img in enumerate((blanked, filled, patches[idx])):
strip[r * (SIDE + GAP):r * (SIDE + GAP) + SIDE, col:col + SIDE] = np.clip(img, 0, 1)
plt.imsave(OUT_DIR / "pretraining_patches.png", strip.repeat(3, axis=0).repeat(3, axis=1))
fig, axes = plt.subplots(1, 3, figsize=(14, 3.8))
ax = axes[0]
steps = [s for s, _, _ in loss_log]
ax.plot(steps, [tr for _, tr, _ in loss_log], "-", color="black", label="training half")
ax.plot(steps, [te for _, _, te in loss_log], "--", color="0.5", label="test half")
ax.axhline(mse_ring, color="black", linestyle=":", label="mean color outside the blank (test)")
ax.set_yscale("log")
ax.set_xlabel("Adam iteration")
ax.set_ylabel("mean squared error of the center color")
ax.set_title("Color pretext task: predict the blanked center")
ax.legend(frameon=False, fontsize=8)
ax = axes[1]
ax.imshow(strip)
ax.set_xticks([])
ax.set_yticks([SIDE // 2 + r * (SIDE + GAP) for r in range(3)])
ax.set_yticklabels(["blanked", "predicted", "original"], fontsize=8)
ax.set_xlabel("eight test patches of 25 m")
ax.set_ylabel("row")
ax.set_title("Center color predicted from the surroundings")
ax = axes[2]
styles = {"mean color": ("^:", "0.5"), "raw pixels": ("s--", "0.5"),
"untrained layers": ("o-", "0.5"), "color-pretrained layers": ("D-", "black"),
"rotation-pretrained layers": ("v-", "black")}
for name, (style, color) in styles.items():
ax.semilogx(SIZES, acc[name], style, color=color, markersize=4, label=name)
ax.axhline(acc_base, color="black", linestyle=":", linewidth=0.8, label="majority baseline")
ax.set_xlabel("number of labeled patches")
ax.set_ylabel("test accuracy (eastern half)")
ax.set_title("Linear classifier: vineyard vs. no vineyard")
ax.legend(frameon=False, fontsize=7, loc="lower right")
fig.tight_layout()
fig.savefig(OUT_DIR / "pretraining.png", dpi=150)
plt.close(fig)
check("[B-plot] the CSV files, the scene and the patch strip were written",
all((OUT_DIR / f).exists() for f in
("pretraining_loss.csv", "pretraining_accuracy.csv",
"pretraining_scene.png", "pretraining_patches.png")))
passed = sum(1 for _, ok in report if ok)
print(f"\n{passed}/{len(report)} checks pass")
[ok] [B-plot] the CSV files, the scene and the patch strip were written 10/10 checks pass
B-plot writes when the script runs