B-weather writes when the script runsDictionary of Applied Machine Learning · fine-tuning
Updated on 2026-11-03
Numerical companion to the entry fine-tuning: it recomputes what the entry states and prints one line per check
The entry's claims, measured: a linear hypothesis is pretrained on a large dataset from a related task, then adapted to a small task-specific training set by a few GD steps. Warm-starting at the pretrained model parameters keeps the iterates in a small neighborhood and reaches a small validation error; the same iteration from a fresh initialization overfits the small training set. Self-contained (numpy/matplotlib only), deterministic.
Run it without installing anything:uv run https://dictionaryofml.org/terms/finetuning.py
uv downloads this script and the pinned NumPy and Matplotlib it needs, then runs it; the script fetches any input file it uses. To keep the output files, download finetuning.py into a folder and run uv run finetuning.py there. With NumPy and Matplotlib already installed, python3 finetuning.py, from any directory — it writes its output files into the current directory. Fixed seeds, so the printed numbers reproduce exactly. Download finetuning.py · Notebook · Open in Colab
One cell per block of the script: the code, and what that code printed when it last ran here
"""
finetuning.py -- numerical companion to the entry 'fine-tuning'.
The entry's claims, measured: a linear hypothesis is pretrained on a large
dataset from a related task, then adapted to a small task-specific training
set by a few GD steps. Warm-starting at the pretrained model
parameters keeps the iterates in a small neighborhood and reaches a small
validation error; the same iteration from a fresh initialization overfits
the small training set. Self-contained (numpy/matplotlib only),
deterministic.
Blocks
------
[B-tasks] Two related tasks with 30 model parameters: pretraining data
(2000 data points) from parameters w*, task data from
w* + delta with a small shift delta. Pretraining recovers
parameters close to w*.
[B-warm] Fine-tuning: GD on the task's small training
set (15 data points), warm-started at the pretrained
parameters, dips to a validation error below twice the noise
level within the first ~20 steps; running all 500 steps
interpolates the 15 points and drifts upward again -- the
measured reason fine-tuning stops early.
[B-fresh] The same iteration from a fresh (zero) initialization drives
the training error low but its validation error stays several
times larger: it overfits the 15 data points.
[B-distance] The iterates stay near where they start: the distance moved
is bounded by the accumulated update lengths eta*sum|grad|,
and hence by eta*T*G with G the largest gradient norm, so T
steps of step size eta stay in the ball of radius eta*T*G
around the initialization. The warm start moves only about
the length of the task shift delta, far less than the fresh
run.
[B-weather] The entry's opening figure: 360 days of synthetic weather at
Krems an der Donau (morning minimum and maximum daytime
temperature; the daily range depends on the month). A linear
hypothesis map is pretrained on all 360 days; for January,
May and November it is fine-tuned by 30 GD steps on that
month's 30 days. Each fine-tuned line lowers the month's
average squared error below the pretrained line's, and moves
the model parameters by less than 1.5.
Outputs
-------
finetuning_days.csv : tmin, tmax, month for the 360 days.
finetuning_days20.csv : every 18th of the 360 days, drawn in panel (a).
finetuning_months15.csv : every other day of January, May and November
(15 each), drawn in panels (b)-(d).
finetuning_lines.csv : tmin grid with the pretrained line (pre) and
the fine-tuned lines (jan, may, nov).
finetuning.png : preview of the opening figure (checking only).
finetuning_curves.png : preview of the validation/distance curves of
[B-warm], [B-fresh], [B-distance] (checking
only).
"""
# /// script
# requires-python = ">=3.10"
# dependencies = ["matplotlib==3.10.7", "numpy==2.3.4"]
# ///
import numpy as np
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
from pathlib import Path
OUT_DIR = Path(__file__).parent
report = []
def check(name, ok):
report.append((name, bool(ok)))
print(f" [{'ok' if ok else 'FAIL'}] {name}")
D = 30
NOISE = 0.5
gen = np.random.default_rng(20260829)
w_star = gen.normal(size=D)
w_star *= 3.0 / np.linalg.norm(w_star) # |w*| = 3
delta = gen.normal(size=D)
delta *= 0.5 / np.linalg.norm(delta) # small task shift, |delta| = 0.5
w_task = w_star + delta
def draw(n, w, seed):
g = np.random.default_rng(seed)
X = g.normal(size=(n, D))
y = X @ w + g.normal(0.0, NOISE, n)
return X, y
def avg_sqerr(X, y, w):
return float(np.mean((y - X @ w) ** 2))
Two related tasks with 30 model parameters: pretraining data (2000 data points) from parameters w*, task data from w* + delta with a small shift delta. Pretraining recovers parameters close to w*.
print("[B-tasks] a large pretraining dataset and a small task dataset")
Xp, yp = draw(2000, w_star, seed=1)
w_pre = np.linalg.lstsq(Xp, yp, rcond=None)[0]
Xt, yt = draw(15, w_task, seed=2)
Xv, yv = draw(100_000, w_task, seed=3)
print(f" pretraining on {len(yp)} data points: |w_pre - w*| = "
f"{np.linalg.norm(w_pre - w_star):.3f} (|w*| = 3, task shift "
f"|delta| = 0.5)")
print(f" task training set: {len(yt)} data points, {D} model parameters")
check("pretraining recovers parameters close to w*",
np.linalg.norm(w_pre - w_star) < 0.15)
def run_gd(w0, steps=500, eta=0.02):
w = w0.copy()
dist, val, moved, gmax = [], [], 0.0, 0.0
for t in range(steps):
g = -2.0 / len(yt) * Xt.T @ (yt - Xt @ w)
w = w - eta * g
moved += eta * float(np.linalg.norm(g))
gmax = max(gmax, float(np.linalg.norm(g)))
dist.append(float(np.linalg.norm(w - w0)))
val.append(avg_sqerr(Xv, yv, w))
return w, dist, val, moved, gmax
[B-tasks] a large pretraining dataset and a small task dataset
pretraining on 2000 data points: |w_pre - w*| = 0.071 (|w*| = 3, task shift |delta| = 0.5)
task training set: 15 data points, 30 model parameters
[ok] pretraining recovers parameters close to w*
Fine-tuning: GD on the task's small training set (15 data points), warm-started at the pretrained parameters, dips to a validation error below twice the noise level within the first ~20 steps; running all 500 steps interpolates the 15 points and drifts upward again -- the measured reason fine-tuning stops early.
print("\n[B-warm] warm-started GD on the 15 data points")
w_warm, dist_w, val_w, moved_w, gmax_w = run_gd(w_pre)
best = int(np.argmin(val_w))
print(f" validation error dips to {val_w[best]:.3f} at step {best + 1} "
f"(noise level {NOISE**2:.2f}), ends at {val_w[-1]:.3f} after "
f"interpolating the 15 points (training error "
f"{avg_sqerr(Xt, yt, w_warm):.3f})")
check("stopped at its best step, the warm start reaches a validation "
"error below twice the noise level", val_w[best] < 2 * NOISE**2)
check("running on drifts upward again -- the reason fine-tuning is "
"stopped after few steps", val_w[-1] > 1.2 * val_w[best])
[B-warm] warm-started GD on the 15 data points
validation error dips to 0.438 at step 16 (noise level 0.25), ends at 0.567 after interpolating the 15 points (training error 0.000)
[ok] stopped at its best step, the warm start reaches a validation error below twice the noise level
[ok] running on drifts upward again -- the reason fine-tuning is stopped after few steps
The same iteration from a fresh (zero) initialization drives the training error low but its validation error stays several times larger: it overfits the 15 data points.
print("\n[B-fresh] the same iteration from a fresh initialization")
w_fresh, dist_f, val_f, moved_f, gmax_f = run_gd(np.zeros(D))
print(f" validation error {val_f[-1]:.3f}; training error "
f"{avg_sqerr(Xt, yt, w_fresh):.3f} (15 data points, 30 parameters)")
check("the fresh run drives the training error below the noise level",
avg_sqerr(Xt, yt, w_fresh) < NOISE**2)
check("yet its validation error is at least four times the warm start's",
val_f[-1] > 4 * val_w[-1])
[B-fresh] the same iteration from a fresh initialization
validation error 2.464; training error 0.000 (15 data points, 30 parameters)
[ok] the fresh run drives the training error below the noise level
[ok] yet its validation error is at least four times the warm start's
The iterates stay near where they start: the distance moved is bounded by the accumulated update lengths eta*sum|grad|, and hence by eta*T*G with G the largest gradient norm, so T steps of step size eta stay in the ball of radius eta*T*G around the initialization. The warm start moves only about the length of the task shift delta, far less than the fresh run.
print("\n[B-distance] the iterates stay near where they start")
print(f" distance moved: warm {dist_w[-1]:.2f} (bound "
f"eta*sum|grad| = {moved_w:.2f}), fresh {dist_f[-1]:.2f}; "
f"task shift |delta| = 0.5, |w*| = 3")
check("the distance moved is bounded by the accumulated step lengths",
dist_w[-1] <= moved_w + 1e-9 and dist_f[-1] <= moved_f + 1e-9)
check("the warm start moves on the scale of the task shift, far less "
"than the fresh run", dist_w[-1] < 1.0 and dist_f[-1] > 2.0)
ETA, T_FT = 0.02, best + 1
ball = ETA * T_FT * gmax_w
print(f" after T = {T_FT} warm-started steps of step size eta = {ETA}: "
f"distance {dist_w[best]:.2f}, ball radius eta*T*G = {ball:.2f} "
f"(largest gradient norm G = {gmax_w:.2f})")
check("the fine-tuned parameters lie in the ball of radius eta*T*G "
"around the pretrained ones", dist_w[best] <= ball + 1e-9)
[B-distance] the iterates stay near where they start
distance moved: warm 0.64 (bound eta*sum|grad| = 0.75), fresh 2.60; task shift |delta| = 0.5, |w*| = 3
[ok] the distance moved is bounded by the accumulated step lengths
[ok] the warm start moves on the scale of the task shift, far less than the fresh run
after T = 16 warm-started steps of step size eta = 0.02: distance 0.32, ball radius eta*T*G = 0.72 (largest gradient norm G = 2.24)
[ok] the fine-tuned parameters lie in the ball of radius eta*T*G around the pretrained ones
The entry's opening figure: 360 days of synthetic weather at Krems an der Donau (morning minimum and maximum daytime temperature; the daily range depends on the month). A linear hypothesis map is pretrained on all 360 days; for January, May and November it is fine-tuned by 30 GD steps on that month's 30 days. Each fine-tuned line lowers the month's average squared error below the pretrained line's, and moves the model parameters by less than 1.5.
print("\n[B-weather] a linear hypothesis map pretrained on a year, fine-tuned "
"to a month")
gw = np.random.default_rng(7)
MONTHS = ["jan", "feb", "mar", "apr", "may", "jun", "jul", "aug", "sep",
"oct", "nov", "dec"]
mean_tmin = [-4, -3, 1, 5, 10, 13, 15, 15, 11, 6, 1, -3] # degrees C
day_range = [2, 4, 7, 9, 12, 12, 12, 11, 9, 7, 6, 3] # tmax - tmin
slope_in_month = [0.9, 0.8, 0.6, 0.4, 0.2, 0.2, 0.2, 0.3, 0.5, 0.7, 1.0, 0.9]
tmin, tmax, month = [], [], []
for m in range(12):
t0 = mean_tmin[m] + gw.normal(0.0, 3.0, 30)
tmin += list(t0)
tmax += list(t0 + day_range[m] + slope_in_month[m] * (t0 - mean_tmin[m])
+ gw.normal(0.0, 1.2, 30))
month += [m + 1] * 30
tmin, tmax, month = np.array(tmin), np.array(tmax), np.array(month)
Xw = np.c_[tmin, np.ones(360)]
w_year = np.linalg.lstsq(Xw, tmax, rcond=None)[0] # pretrained line
def month_gd(m, steps=30, eta=0.005):
sel = month == m
Xm, ym = Xw[sel], tmax[sel]
w = w_year.copy()
for _ in range(steps):
w = w - eta * (-2.0 / len(ym) * Xm.T @ (ym - Xm @ w))
return w, avg_sqerr(Xm, ym, w_year), avg_sqerr(Xm, ym, w)
lines = {"pre": w_year}
ok_loss, ok_dist = True, True
for name, m in (("jan", 1), ("may", 5), ("nov", 11)):
w_m, before, after = month_gd(m)
lines[name] = w_m
d = float(np.linalg.norm(w_m - w_year))
print(f" {name}: average squared error {before:.2f} (pretrained) -> "
f"{after:.2f} (fine-tuned), parameters moved {d:.2f}")
ok_loss &= after < before
ok_dist &= d < 1.5
check("fine-tuning lowers each month's average squared error below the "
"pretrained line's", ok_loss)
check("each fine-tuned line stays within distance 1.5 of the pretrained "
"parameters", ok_dist)
with open(OUT_DIR / "finetuning_days.csv", "w") as fh:
fh.write("tmin,tmax,month\n")
for a, b, c in zip(tmin, tmax, month):
fh.write(f"{a:.2f},{b:.2f},{c}\n")
# the entry's figure draws 20 of the 360 days (every 18th day) in the
# pretraining panel and every other day of a month (15 of 30) in the
# fine-tuning panels, so the lines stay visible; the fits use all days
with open(OUT_DIR / "finetuning_days20.csv", "w") as fh:
fh.write("tmin,tmax\n")
for i in range(0, 360, 18):
fh.write(f"{tmin[i]:.2f},{tmax[i]:.2f}\n")
with open(OUT_DIR / "finetuning_months15.csv", "w") as fh:
fh.write("tmin,tmax,month\n")
for m in (1, 5, 11):
sel = np.flatnonzero(month == m)[::2]
for i in sel:
fh.write(f"{tmin[i]:.2f},{tmax[i]:.2f},{m}\n")
check("the drawn subsets hold 20 of the 360 days and 15 of each month's 30",
len(range(0, 360, 18)) == 20
and all(np.flatnonzero(month == m)[::2].size == 15 for m in (1, 5, 11)))
tg = np.linspace(-12.0, 22.0, 2)
with open(OUT_DIR / "finetuning_lines.csv", "w") as fh:
fh.write("tmin,pre,jan,may,nov\n")
for a in tg:
fh.write(f"{a:.2f}," + ",".join(f"{lines[k][0] * a + lines[k][1]:.3f}"
for k in ("pre", "jan", "may", "nov")) + "\n")
# --------------------------------------------------------------- preview
fig, axs = plt.subplots(2, 3, figsize=(9.6, 6.0))
top = axs[0, 1]
for a in (axs[0, 0], axs[0, 2]):
a.axis("off")
top.plot(tmin, tmax, "o", ms=2.5, color="0.45", label="360 days")
top.plot(tg, lines["pre"][0] * tg + lines["pre"][1], "--", color="0.5",
lw=1.5, label="pretrained line")
top.set_xlabel("morning minimum temperature")
top.set_ylabel("maximum daytime temperature")
top.set_title("pretraining on the whole year", fontsize=9)
top.legend(frameon=False, fontsize=8)
for a, (name, m, label) in zip(axs[1], (("jan", 1, "January"), ("may", 5, "May"),
("nov", 11, "November"))):
sel = month == m
a.plot(tmin[sel], tmax[sel], "ko", ms=3, label=f"{label}, 30 days")
a.plot(tg, lines["pre"][0] * tg + lines["pre"][1], "--", color="0.5",
lw=1.2, label="pretrained line")
a.plot(tg, lines[name][0] * tg + lines[name][1], "k-", lw=1.5,
label="fine-tuned line")
a.set_xlabel("morning minimum temperature")
a.set_ylabel("maximum daytime temperature")
a.set_title(f"fine-tuning to {label}", fontsize=9)
a.set_xlim(-13, 15) # one temperature range for all three panels
a.set_ylim(-18, 29)
a.legend(frameon=False, fontsize=7)
fig.tight_layout()
fig.savefig(OUT_DIR / "finetuning.png", dpi=110)
fig, ax = plt.subplots(1, 2, figsize=(9.2, 3.6))
steps = range(1, len(val_w) + 1)
ax[0].plot(steps, val_w, "-", color="black", label="warm start (pretrained)")
ax[0].plot(steps, val_f, "--", color="0.4", label="fresh initialization")
ax[0].axhline(NOISE**2, color="0.75", lw=0.8)
ax[0].set_yscale("log")
ax[0].set_xlabel("GD step")
ax[0].set_ylabel("validation error")
ax[0].set_title("[B-warm]/[B-fresh] 15 data points, 30 parameters",
fontsize=9)
ax[0].legend(frameon=False, fontsize=8)
ax[1].plot(steps, dist_w, "-", color="black", label="warm start")
ax[1].plot(steps, dist_f, "--", color="0.4", label="fresh initialization")
ax[1].set_xlabel("GD step")
ax[1].set_ylabel("distance from initialization")
ax[1].set_title("[B-distance] how far the iterates move", fontsize=9)
ax[1].legend(frameon=False, fontsize=8)
fig.tight_layout()
fig.savefig(OUT_DIR / "finetuning_curves.png", dpi=110)
n_ok = sum(ok for _, ok in report)
print(f"\n{n_ok}/{len(report)} checks pass")
print("wrote finetuning_days.csv, finetuning_lines.csv, finetuning.png, "
"finetuning_curves.png")
if n_ok != len(report):
raise SystemExit(1)
[B-weather] a linear hypothesis map pretrained on a year, fine-tuned to a month
jan: average squared error 4.10 (pretrained) -> 1.16 (fine-tuned), parameters moved 0.29
may: average squared error 7.81 (pretrained) -> 3.15 (fine-tuned), parameters moved 0.25
nov: average squared error 3.67 (pretrained) -> 0.98 (fine-tuned), parameters moved 0.48
[ok] fine-tuning lowers each month's average squared error below the pretrained line's
[ok] each fine-tuned line stays within distance 1.5 of the pretrained parameters
[ok] the drawn subsets hold 20 of the 360 days and 15 of each month's 30
11/11 checks pass
wrote finetuning_days.csv, finetuning_lines.csv, finetuning.png, finetuning_curves.png
B-weather writes when the script runs