P-transfer writes when the script runsDictionary of Applied Machine Learning · self-supervised learning
Numerical companion to the entry self-supervised learning: it recomputes what the entry states and prints one line per check
One block per paragraph of the entry (marked [P...]): each block verifies numerically what the corresponding statement asserts. Self-contained (numpy/matplotlib only), fixed seed.
Run it with python3 selfsupervisedlearning.py, from any directory — it writes its output files into the current directory. Requires NumPy and Matplotlib only, and uses fixed seeds, so the printed numbers reproduce exactly. Download selfsupervisedlearning.py · Notebook · Open in Colab
One cell per block of the script: the code, and what that code printed when it last ran here
"""
selfsupervisedlearning.py — numerical companion to the entry
'self-supervised learning'.
One block per paragraph of the entry (marked [P...]): each block verifies
numerically what the corresponding statement asserts. Self-contained
(numpy/matplotlib only), fixed seed.
Blocks
------
[P-construct] Withholding some features of a data point and using them as
its label turns an unlabeled collection into a training set:
a stream of m tokens, each predicted from the n before it,
yields m - n labeled data points, none of them annotated.
Withholding something else yields a different training set
from the same collection.
[P-nlp] The next token is predictable from the ones before it: a
map fitted by ERM on the constructed labels gets a
larger fraction of held-out tokens right than the map
that always answers with the most frequent token.
[P-vision] The same for pixels: three quarters of an image's patches are
deleted and their pixel values predicted from the quarter
left visible, which beats the average image by more than ten
times in squared error,
so the constructed task is one that ERM can learn.
[P-compose] The fitted map factors as h = s . phi. Fitting it shapes phi
even though phi appears nowhere in the loss: the pretext map,
which never sees a label, has range aligned with the latent
structure to within canonical cosines of 0.99.
[P-transfer] phi is what is carried over. s is dropped and a small
replacement fitted on phi's output: with only 10 labeled data
points that beats the same rule fitted on the raw features.
Outputs
-------
selfsupervisedlearning.png : preview figure (checking only).
"""
import numpy as np
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
from pathlib import Path
OUT_DIR = Path(__file__).parent
rng = np.random.default_rng(20260826)
report = []
def check(name, ok):
report.append((name, ok))
print(f" [{'ok' if ok else 'FAIL'}] {name}")
Withholding some features of a data point and using them as its label turns an unlabeled collection into a training set: a stream of m tokens, each predicted from the n before it, yields m - n labeled data points, none of them annotated. Withholding something else yields a different training set from the same collection.
# An unlabeled stream of tokens. Withhold the next token: every position
# supplies one labeled data point, and nothing was annotated by hand.
print("[P-construct] withholding a feature turns raw data into a training set")
VOCAB, CORPUS = 6, 20000
trans = rng.random((VOCAB, VOCAB)) ** 3 # peaked rows: real structure
trans /= trans.sum(axis=1, keepdims=True)
stream = np.empty(CORPUS, dtype=int)
stream[0] = 0
for t in range(1, CORPUS):
stream[t] = rng.choice(VOCAB, p=trans[stream[t - 1]])
CONTEXT = 2
feats = np.stack([stream[i:CORPUS - CONTEXT + i] for i in range(CONTEXT)], axis=1)
labels = stream[CONTEXT:]
check(f"a stream of {CORPUS} tokens yields {CORPUS - CONTEXT} labeled data points",
len(labels) == CORPUS - CONTEXT and len(feats) == len(labels))
check("every label is a token taken from the stream itself",
bool(np.all(labels == stream[CONTEXT:])))
# withholding the PREVIOUS token instead gives a different training set
labels_back = stream[:-CONTEXT]
check("withholding a different feature gives a different training set",
not np.array_equal(labels, labels_back))
[P-construct] withholding a feature turns raw data into a training set [ok] a stream of 20000 tokens yields 19998 labeled data points [ok] every label is a token taken from the stream itself [ok] withholding a different feature gives a different training set
The next token is predictable from the ones before it: a map fitted by ERM on the constructed labels gets a larger fraction of held-out tokens right than the map that always answers with the most frequent token.
# ERM on the constructed labels: per context, answer with the token that
# followed it most often in the training half.
print("\n[P-nlp] the next token is predictable from the ones before it")
split = len(labels) // 2
ctx_id = feats[:, 0] * VOCAB + feats[:, 1]
counts = np.zeros((VOCAB * VOCAB, VOCAB))
np.add.at(counts, (ctx_id[:split], labels[:split]), 1)
table = counts.argmax(axis=1)
most_frequent = np.bincount(labels[:split], minlength=VOCAB).argmax()
hit_ctx = float(np.mean(table[ctx_id[split:]] == labels[split:]))
hit_const = float(np.mean(labels[split:] == most_frequent))
print(f" held-out tokens predicted correctly: context {hit_ctx:.3f}, "
f"most-frequent token {hit_const:.3f}")
check("the context-based map beats the constant one",
hit_ctx > hit_const + 0.01)
check("neither map used a hand-annotated label", True)
# the masked variant: withhold a token in the MIDDLE and predict it from
# both sides -- the same construction with a different withheld feature
mid_ctx = stream[:-2] * VOCAB + stream[2:] # (before, after)
mid_lab = stream[1:-1]
msplit = len(mid_lab) // 2
mcounts = np.zeros((VOCAB * VOCAB, VOCAB))
np.add.at(mcounts, (mid_ctx[:msplit], mid_lab[:msplit]), 1)
mtable = mcounts.argmax(axis=1)
hit_mid = float(np.mean(mtable[mid_ctx[msplit:]] == mid_lab[msplit:]))
print(f" a token withheld in the middle, predicted from both sides: "
f"{hit_mid:.3f}")
check("withholding a middle token also beats the constant map",
hit_mid > hit_const + 0.01)
[P-nlp] the next token is predictable from the ones before it
held-out tokens predicted correctly: context 0.576, most-frequent token 0.300
[ok] the context-based map beats the constant one
[ok] neither map used a hand-annotated label
a token withheld in the middle, predicted from both sides: 0.751
[ok] withholding a middle token also beats the constant map
The same for pixels: three quarters of an image's patches are deleted and their pixel values predicted from the quarter left visible, which beats the average image by more than ten times in squared error, so the constructed task is one that ERM can learn.
# Images with structure across neighbouring pixels, split into patches.
# Three quarters of the patches are deleted and their pixel values are
# predicted from the quarter left visible. The mask is the same for every
# image here, which is a simplification: MAE redraws it per image.
print("\n[P-vision] deleted patches are predictable from the patches left")
SIDE, PATCH, NIMG = 16, 4, 800
gy, gx = np.mgrid[0:SIDE, 0:SIDE] / SIDE
images = np.empty((NIMG, SIDE, SIDE))
for i in range(NIMG):
fx, fy, ph = rng.uniform(1, 3), rng.uniform(1, 3), rng.uniform(0, 6.28)
images[i] = np.sin(2 * np.pi * (fx * gx + fy * gy) + ph)
images += 0.05 * rng.normal(size=images.shape)
npatch = (SIDE // PATCH) ** 2
keep = np.zeros(npatch, dtype=bool)
keep[rng.choice(npatch, size=npatch // 4, replace=False)] = True
check(f"{100 * (1 - keep.mean()):.0f}% of the {npatch} patches are deleted",
np.isclose(keep.mean(), 0.25))
pix = np.zeros((SIDE, SIDE), dtype=bool) # visible pixel mask
for q in range(npatch):
pr, pc = divmod(q, SIDE // PATCH)
pix[pr * PATCH:(pr + 1) * PATCH, pc * PATCH:(pc + 1) * PATCH] = keep[q]
flat = images.reshape(NIMG, -1)
vis_px, hid_px = pix.ravel(), ~pix.ravel()
ntr = NIMG // 2
# ERM with the squared error: hidden pixel values from the visible ones
Wv, *_ = np.linalg.lstsq(flat[:ntr][:, vis_px], flat[:ntr][:, hid_px],
rcond=None)
err_vis = float(np.mean((flat[ntr:][:, vis_px] @ Wv - flat[ntr:][:, hid_px]) ** 2))
# the alternative that uses nothing about the particular image
avg = flat[:ntr][:, hid_px].mean(axis=0)
err_avg = float(np.mean((avg - flat[ntr:][:, hid_px]) ** 2))
print(f" squared error on held-out images: from the visible patches "
f"{err_vis:.4f}, from the average image {err_avg:.4f}")
check("predicting deleted patches from the visible ones beats the average",
err_vis < 0.5 * err_avg)
[P-vision] deleted patches are predictable from the patches left
[ok] 75% of the 16 patches are deleted
squared error on held-out images: from the visible patches 0.0407, from the average image 0.5046
[ok] predicting deleted patches from the visible ones beats the average
The fitted map factors as h = s . phi. Fitting it shapes phi even though phi appears nowhere in the loss: the pretext map, which never sees a label, has range aligned with the latent structure to within canonical cosines of 0.99.
# The map built for the constructed task factors as h = s . phi:
# phi maps a data point to a representation, s maps that to the withheld
# feature. Only the composition appears in the loss, yet fitting it is what
# shapes phi.
print("\n[P-compose] fitting the composition shapes phi, which the loss never mentions")
DIM, LATENT, NUN, NTEST = 40, 3, 4000, 4000
A = rng.normal(size=(DIM, LATENT))
def draw(n):
z = rng.normal(size=(n, LATENT))
return z @ A.T + 0.30 * rng.normal(size=(n, DIM)), z
vis, hid = slice(0, DIM // 2), slice(DIM // 2, DIM)
Xun, _ = draw(NUN) # unlabeled: no z used
# the constructed task: predict the withheld features from the visible ones
Wpre, *_ = np.linalg.lstsq(Xun[:, vis], Xun[:, hid], rcond=None)
# Wpre sends visible coordinates to hidden ones, so the directions acting ON
# the visible features are its LEFT factors. The right ones live in the hidden
# coordinate space; projecting visible features onto them is meaningless.
U, S, _ = np.linalg.svd(Wpre, full_matrices=False)
phi = U[:, :LATENT] # the representation
check(f"the pretext map has {LATENT} directions that matter "
f"(gap {S[LATENT - 1]:.2f} to {S[LATENT]:.2f})", S[LATENT - 1] > 3 * S[LATENT])
Q, _ = np.linalg.qr(A[vis, :]) # the latent structure, seen in x_vis
cos = np.linalg.svd(phi.T @ Q, compute_uv=False)
print(f" canonical cosines between range(phi) and the latent structure: "
f"{np.round(cos, 3)}")
check("phi recovers the latent structure it was never told about",
cos.min() > 0.9)
[P-compose] fitting the composition shapes phi, which the loss never mentions
[ok] the pretext map has 3 directions that matter (gap 0.72 to 0.13)
canonical cosines between range(phi) and the latent structure: [0.998 0.998 0.996]
[ok] phi recovers the latent structure it was never told about
phi is what is carried over. s is dropped and a small replacement fitted on phi's output: with only 10 labeled data points that beats the same rule fitted on the raw features.
# s is dropped; phi is kept and a small replacement fitted on its output,
# using the few labels the later task has.
print("\n[P-transfer] phi is what is carried over, and it is what pays")
def fit_predict(feat_tr, lab_tr, feat_te):
"""Fit by minimizing the average squared error, then take the sign."""
F = np.hstack([feat_tr, np.ones((len(feat_tr), 1))])
w, *_ = np.linalg.lstsq(F, lab_tr, rcond=None)
G = np.hstack([feat_te, np.ones((len(feat_te), 1))])
return np.sign(G @ w)
NLAB = 10
Xte, zte = draw(NTEST)
yte = np.sign(zte[:, 0])
raw_hits, rep_hits = [], []
for _ in range(200): # many small labeled sets
Xtr, ztr = draw(NLAB)
ytr = np.sign(ztr[:, 0])
raw_hits.append(np.mean(fit_predict(Xtr[:, vis], ytr, Xte[:, vis]) == yte))
rep_hits.append(np.mean(fit_predict(Xtr[:, vis] @ phi, ytr,
Xte[:, vis] @ phi) == yte))
raw, repr_ = float(np.mean(raw_hits)), float(np.mean(rep_hits))
print(f" held-out labels predicted correctly with {NLAB} labeled data "
f"points: raw features {raw:.3f}, representation {repr_:.3f}")
check("the representation beats the raw features when labels are scarce",
repr_ > raw + 0.05)
check("the pretext task never saw a label", True)
# ------------------------------------------------------------ preview
fig, ax = plt.subplots(1, 2, figsize=(9, 3.2))
ax[0].bar([0, 1], [hit_const, hit_ctx], width=0.5,
color=["0.75", "0.35"], edgecolor="black")
ax[0].set_xticks([0, 1])
ax[0].set_xticklabels(["most frequent\ntoken", "from the\ncontext"])
ax[0].set_ylabel("fraction of held-out tokens right")
ax[0].set_xlabel("fitted map")
ax[0].set_title("[P-nlp] the next token is predictable")
ax[1].bar([0, 1], [raw, repr_], width=0.5,
color=["0.75", "0.35"], edgecolor="black")
ax[1].set_xticks([0, 1])
ax[1].set_xticklabels(["raw\nfeatures", "learned\nrepresentation"])
ax[1].set_ylabel("fraction of held-out labels right")
ax[1].set_xlabel(f"features used, with {NLAB} labeled data points")
ax[1].set_title("[P-transfer] what the pretext task leaves behind")
fig.tight_layout()
fig.savefig(OUT_DIR / "selfsupervisedlearning.png", dpi=110)
print(f"\n{sum(ok for _, ok in report)}/{len(report)} checks passed")
assert all(ok for _, ok in report)
[P-transfer] phi is what is carried over, and it is what pays
held-out labels predicted correctly with 10 labeled data points: raw features 0.778, representation 0.848
[ok] the representation beats the raw features when labels are scarce
[ok] the pretext task never saw a label
12/12 checks passed
P-transfer writes when the script runs