Dictionary of Applied Machine Learning · encoder
Updated on 2026-11-20
Numerical companion to the entry encoder: it recomputes what the entry states and prints one line per check
Numerical companion to the glossary entry 'encoder'. The GeoSphere Austria weather station Krems an der Donau (station id 3805) records eight measurements per day. The principal component analysis (PCA) encoder is the matrix whose rows are the eigenvectors of the sample covariance matrix belonging to its largest eigenvalues; it maps the eight scaled measurements of a day to a code of d' numbers, and the decoder, its transpose, maps the code back to eight numbers.
Run it without installing anything:uv run https://dictionaryofml.org/terms/encoder.py
uv downloads this script and the pinned NumPy and Matplotlib it needs, then runs it; the script fetches any input file it uses. To keep the output files, download encoder.py into a folder and run uv run encoder.py there. With NumPy and Matplotlib already installed, python3 encoder.py, from any directory — it writes its output files into the current directory. Fixed seeds, so the printed numbers reproduce exactly. Download encoder.py · Notebook · Open in Colab
Before running a downloaded copy, scan it: uvx bandit -q -t B102,B307,B602,B605 encoder.py (B102,B307,B602,B605 are the exec, eval and shell checks of bandit) reports nothing: the script runs no shell command and evaluates no generated code.
One cell per block of the script: the code, and what that code printed when it last ran here
"""Encoders on a year of days at Krems an der Donau and on an excerpt of
the Universal Declaration of Human Rights: a map from the raw features of
a data point to a shorter code, judged by how well a decoder rebuilds the
data point from that code.
Purpose
-------
Numerical companion to the glossary entry 'encoder'. The GeoSphere
Austria weather station Krems an der Donau (station id 3805) records
eight measurements per day. The principal component analysis (PCA)
encoder is the matrix whose rows are the eigenvectors of the sample
covariance matrix belonging to its largest eigenvalues; it maps the eight
scaled measurements of a day to a code of d' numbers, and the decoder,
its transpose, maps the code back to eight numbers.
The demo recomputes what the entry claims. (1) Encoder followed by
decoder returns approximately the day it started from: with a code of two
numbers the eight measurements of 1 January 2024 come back to within
0.80 on the scaled axes, and the reconstruction error averaged over the
366 days is 2.91 of the 8.00 total variance. (2) The reconstruction
error falls with the code length d' and equals the sum of the eigenvalues
the code drops, down to zero at d' = 8. (3) The encoder matters, not
only the code length: a fixed random orthonormal encoder leaves a larger
reconstruction error at every code length below 8. (4) The token-to-
vector step of a language model is an encoder of the same shape -- the
one-hot vector of a token times the embedding matrix is the token's row
of that matrix -- and here it loses nothing, since distinct rows let the
nearest row recover every token. (5) What the code of two numbers drops
is beyond every decoder and can matter for a later learning task: it
keeps 96 percent of the sample variance of the mean temperature and 3
percent of the wind speed, two days with almost the same code differ by
3.5 meters per second in wind speed while the decoder returns one value
for both, a linear classifier learned by least squares calls windy days
with an accuracy of 66 percent from the code against 98 percent from
the eight measurements, and one new feature chosen from the labels, the
linear discriminant analysis direction, reaches 98 percent as well.
Deterministic: the weather data are a fixed archive year, PCA is an
eigenvalue decomposition, and the random encoder and the embedding matrix
are drawn from fixed seeds. Self-contained: numpy and matplotlib only
(stdlib urllib for the download, which falls back to the committed CSV).
Blocks
------
[B-data] Download the 366 days of 2024 with eight measurements each and
scale every measurement to zero sample mean and unit sample
variance, so that the eight carry comparable weight.
[B-encoder] The PCA encoder for a code of two numbers: the rows of the
matrix are eigenvectors of the sample covariance matrix, the
code of a day is the matrix times its scaled measurements, and
the decoder is the transpose. Encoder then decoder returns the
day approximately; the entry's named day is reported.
[B-length] The reconstruction error against the code length d' = 1 to 8:
it equals the sum of the dropped eigenvalues and vanishes at
d' = 8, already at d' = 7 because the mean temperature is the
midpoint of the lowest and the highest, and a fixed random
orthonormal encoder leaves more at every code length below 8.
[B-token] The token-to-vector encoder of a language model on an excerpt
of the Universal Declaration of Human Rights: a one-hot vector
of length |V| times the embedding matrix selects a row, and the
nearest row recovers every token, so this encoder loses nothing.
[B-drops] What the code of two numbers drops: the kept fraction of each
measurement, a pair of days with almost the same code and very
different wind speeds, the accuracy of a linear classifier for
windy days from the code against the eight measurements and
against one linear discriminant analysis feature, and the error
an encoder learned on the cold half leaves on the warm half.
[B-plot] Preview: the two-number code of 21 days, and the reconstruction
error against the code length.
Outputs
-------
encoder_weather.csv : date and the eight measurements (committed copy)
encoder_code.csv : date, z1, z2, warm -- the code of all 366 days
encoder_code_plot.csv : z1, z2, warm, month -- every 18th of those days,
the 21 dots of the figure
encoder_errors.csv : dprime, pca, random -- error against code length
encoder.png : matplotlib preview (checking only)
"""
# /// script
# requires-python = ">=3.10"
# dependencies = ["matplotlib==3.10.7", "numpy==2.3.4"]
# ///
import json
import urllib.request
from pathlib import Path
import numpy as np
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
OUT_DIR = Path(__file__).parent
report = []
def check(name, ok):
report.append((name, bool(ok)))
print(f" [{'ok' if ok else 'FAIL'}] {name}")
Installed 11 packages in 84ms
Download the 366 days of 2024 with eight measurements each and scale every measurement to zero sample mean and unit sample variance, so that the eight carry comparable weight.
print("[B-data] eight daily measurements at Krems an der Donau, 2024")
PARAMS = ["tlmin", "tlmax", "tl_mittel", "rr", "so_h", "rf_mittel",
"p_mittel", "vv_mittel"]
URL = ("https://dataset.api.hub.geosphere.at/v1/station/historical/"
f"klima-v2-1d?parameters={','.join(PARAMS)}&station_ids=3805"
"&start=2024-01-01&end=2024-12-31")
CSV = OUT_DIR / "encoder_weather.csv"
try:
with urllib.request.urlopen(URL, timeout=120) as resp:
payload = json.load(resp)
params = payload["features"][0]["properties"]["parameters"]
stamps = [t[:10] for t in payload["timestamps"]]
M = np.stack([np.array(params[p]["data"], dtype=float) for p in PARAMS], 1)
M[:, 3] = np.maximum(M[:, 3], 0.0) # -1 marks a day without rain
with open(CSV, "w") as fh:
fh.write("date," + ",".join(PARAMS) + "\n")
for day, row in zip(stamps, M):
fh.write(day + "," + ",".join(f"{v:g}" for v in row) + "\n")
print(f" downloaded {len(stamps)} days")
except OSError as err:
print(f" download failed ({err}); reading the committed CSV")
rows = CSV.read_text().strip().splitlines()[1:]
stamps = [r.split(",")[0] for r in rows]
M = np.array([[float(v) for v in r.split(",")[1:]] for r in rows])
check("[B-data] 366 days with eight measurements each", M.shape == (366, 8))
check("[B-data] no day is missing a measurement", np.all(np.isfinite(M)))
X = (M - M.mean(axis=0)) / M.std(axis=0)
m, d = X.shape
check(f"[B-data] after scaling, every measurement has sample mean 0 and "
f"sample variance 1, so the {d} together carry a total variance of "
f"{float(np.sum(np.var(X, axis=0))):.2f}",
abs(float(np.sum(np.var(X, axis=0))) - d) < 1e-9)
[B-data] eight daily measurements at Krems an der Donau, 2024
downloaded 366 days
[ok] [B-data] 366 days with eight measurements each
[ok] [B-data] no day is missing a measurement
[ok] [B-data] after scaling, every measurement has sample mean 0 and sample variance 1, so the 8 together carry a total variance of 8.00
The PCA encoder for a code of two numbers: the rows of the matrix are eigenvectors of the sample covariance matrix, the code of a day is the matrix times its scaled measurements, and the decoder is the transpose. Encoder then decoder returns the day approximately; the entry's named day is reported.
print("[B-encoder] the PCA encoder: eigenvectors of the sample covariance")
C = X.T @ X / m # sample covariance matrix
evals, evecs = np.linalg.eigh(C)
order = np.argsort(evals)[::-1]
evals, evecs = evals[order], evecs[:, order]
check(f"[B-encoder] the eight eigenvalues of the sample covariance matrix "
f"sum to the total variance {float(np.sum(evals)):.2f}; the two "
f"largest are {evals[0]:.2f} and {evals[1]:.2f}",
abs(float(np.sum(evals)) - d) < 1e-9)
DPRIME = 2
W = evecs[:, :DPRIME].T # encoder, d' by d
Z = X @ W.T # the code of every day
Xhat = Z @ W # decoder: the transpose
check(f"[B-encoder] the encoder has shape ({DPRIME}, {d}) and its rows are "
f"orthonormal", W.shape == (DPRIME, d)
and np.allclose(W @ W.T, np.eye(DPRIME)))
err2 = float(np.mean(np.sum((X - Xhat) ** 2, axis=1)))
check(f"[B-encoder] averaged over the {m} days, the squared distance between "
f"a day and its reconstruction is {err2:.2f} of the total variance "
f"{d:.2f}, so a code of {DPRIME} numbers keeps "
f"{100 * (1 - err2 / d):.0f} percent of it", 0 < err2 < d)
i0 = stamps.index("2024-01-01")
gap0 = np.abs(X[i0] - Xhat[i0])
check(f"[B-encoder] encoder then decoder returns the day approximately: for "
f"1 January 2024 the code is ({Z[i0, 0]:.2f}, {Z[i0, 1]:.2f}) and the "
f"eight reconstructed measurements differ from the scaled originals "
f"by at most {gap0.max():.2f}", gap0.max() < 1.0)
warm = X[:, 1] > np.median(X[:, 1]) # maximum temperature above median
check(f"[B-encoder] the first code number separates the {int(np.sum(warm))} "
f"days whose maximum temperature is above the median from the "
f"{int(np.sum(~warm))} others: its mean is {Z[warm, 0].mean():.2f} on "
f"the warm days and {Z[~warm, 0].mean():.2f} on the cold ones",
abs(Z[warm, 0].mean() - Z[~warm, 0].mean()) > 1.0)
[B-encoder] the PCA encoder: eigenvectors of the sample covariance [ok] [B-encoder] the eight eigenvalues of the sample covariance matrix sum to the total variance 8.00; the two largest are 3.62 and 1.47 [ok] [B-encoder] the encoder has shape (2, 8) and its rows are orthonormal [ok] [B-encoder] averaged over the 366 days, the squared distance between a day and its reconstruction is 2.91 of the total variance 8.00, so a code of 2 numbers keeps 64 percent of it [ok] [B-encoder] encoder then decoder returns the day approximately: for 1 January 2024 the code is (1.16, -0.25) and the eight reconstructed measurements differ from the scaled originals by at most 0.80 [ok] [B-encoder] the first code number separates the 182 days whose maximum temperature is above the median from the 184 others: its mean is -1.62 on the warm days and 1.60 on the cold ones
The reconstruction error against the code length d' = 1 to 8: it equals the sum of the dropped eigenvalues and vanishes at d' = 8, already at d' = 7 because the mean temperature is the midpoint of the lowest and the highest, and a fixed random orthonormal encoder leaves more at every code length below 8.
print("[B-length] reconstruction error against the code length")
rng = np.random.default_rng(2024)
Q = np.linalg.qr(rng.standard_normal((d, d)))[0] # random orthonormal rows
errs_pca, errs_rnd, dropped = [], [], []
for k in range(1, d + 1):
Wk = evecs[:, :k].T
errs_pca.append(float(np.mean(np.sum((X - (X @ Wk.T) @ Wk) ** 2, axis=1))))
Rk = Q[:, :k].T
errs_rnd.append(float(np.mean(np.sum((X - (X @ Rk.T) @ Rk) ** 2, axis=1))))
dropped.append(float(np.sum(evals[k:])))
check(f"[B-length] the error of the PCA encoder equals the sum of the "
f"eigenvalues the code drops at every code length (largest difference "
f"{max(abs(a - b) for a, b in zip(errs_pca, dropped)):.2e})",
np.allclose(errs_pca, dropped))
check(f"[B-length] the error falls with the code length, from "
f"{errs_pca[0]:.2f} at one number to {errs_pca[1]:.2f} at two and "
f"{errs_pca[3]:.2f} at four, and vanishes at {d}: {errs_pca[-1]:.1e}",
np.all(np.diff(errs_pca) < 0) and errs_pca[-1] < 1e-12)
check(f"[B-length] the smallest eigenvalue is {evals[-1]:.1e}, so one of the "
f"eight measurements adds nothing the other seven do not already "
f"carry: a code of seven numbers leaves {errs_pca[6]:.1e}",
evals[-1] < 1e-3)
check(f"[B-length] the random orthonormal encoder leaves more error at every "
f"code length below {d} ({errs_rnd[1]:.2f} against {errs_pca[1]:.2f} at "
f"two numbers), so the encoder matters and not only the code length",
all(r > p + 1e-9 for r, p in zip(errs_rnd[:-1], errs_pca[:-1])))
[B-length] reconstruction error against the code length [ok] [B-length] the error of the PCA encoder equals the sum of the eigenvalues the code drops at every code length (largest difference 1.24e-14) [ok] [B-length] the error falls with the code length, from 4.38 at one number to 2.91 at two and 0.88 at four, and vanishes at 8: 3.0e-30 [ok] [B-length] the smallest eigenvalue is 6.5e-06, so one of the eight measurements adds nothing the other seven do not already carry: a code of seven numbers leaves 6.5e-06 [ok] [B-length] the random orthonormal encoder leaves more error at every code length below 8 (5.71 against 2.91 at two numbers), so the encoder matters and not only the code length
The token-to-vector encoder of a language model on an excerpt of the Universal Declaration of Human Rights: a one-hot vector of length |V| times the embedding matrix selects a row, and the nearest row recovers every token, so this encoder loses nothing.
print("[B-token] the token-to-vector encoder on a UDHR excerpt")
EXCERPT = ("All human beings are born free and equal in dignity and rights. "
"They are endowed with reason and conscience and should act "
"towards one another in a spirit of brotherhood. Everyone has "
"the right to life, liberty and security of person.")
tokens = EXCERPT.lower().replace(",", "").replace(".", "").split()
vocab = sorted(set(tokens))
nv, dtok = len(vocab), 4
check(f"[B-token] the excerpt has {len(tokens)} tokens over a vocabulary of "
f"{nv} distinct ones, and each is encoded as a vector of {dtok} "
f"numbers", nv < len(tokens) and dtok < nv)
rng_tok = np.random.default_rng(7)
E = rng_tok.standard_normal((nv, dtok)) # the embedding matrix
v = vocab.index("dignity")
onehot = np.zeros(nv)
onehot[v] = 1.0
check(f"[B-token] the one-hot vector of a token times the embedding matrix "
f"is that token's row: for token {v} the two agree to "
f"{float(np.max(np.abs(onehot @ E - E[v]))):.1e}",
np.allclose(onehot @ E, E[v]))
codes = E[[vocab.index(t) for t in tokens]]
recovered = [int(np.argmin(np.sum((E - c) ** 2, axis=1))) for c in codes]
check(f"[B-token] the nearest row recovers the token for all {len(tokens)} "
f"tokens, so this encoder shortens a vector of length {nv} to one of "
f"length {dtok} and still loses nothing",
recovered == [vocab.index(t) for t in tokens])
[B-token] the token-to-vector encoder on a UDHR excerpt [ok] [B-token] the excerpt has 41 tokens over a vocabulary of 34 distinct ones, and each is encoded as a vector of 4 numbers [ok] [B-token] the one-hot vector of a token times the embedding matrix is that token's row: for token 10 the two agree to 0.0e+00 [ok] [B-token] the nearest row recovers the token for all 41 tokens, so this encoder shortens a vector of length 34 to one of length 4 and still loses nothing
What the code of two numbers drops: the kept fraction of each measurement, a pair of days with almost the same code and very different wind speeds, the accuracy of a linear classifier for windy days from the code against the eight measurements and against one linear discriminant analysis feature, and the error an encoder learned on the cold half leaves on the warm half.
print("[B-drops] what the code of two numbers drops")
kept = 1.0 - np.mean((X - Xhat) ** 2, axis=0)
check(f"[B-drops] the code of {DPRIME} numbers does not keep the eight "
f"measurements equally well: of the unit sample variance of each, it "
f"keeps {100 * kept[2]:.0f} percent of the mean temperature and "
f"{100 * kept[7]:.0f} percent of the wind speed",
kept[2] > 0.95 and kept[7] < 0.05)
wind = M[:, 7]
code_dist = np.sqrt(((Z[:, None, :] - Z[None, :, :]) ** 2).sum(axis=2))
np.fill_diagonal(code_dist, np.inf)
twins = [(abs(wind[p] - wind[q]), p, q) for p in range(m)
for q in range(p + 1, m) if code_dist[p, q] < 0.1]
twins.sort(reverse=True)
wind_gap, p0, q0 = twins[0]
rebuilt = Xhat[[p0, q0], 7] * wind.std() + wind.mean()
check(f"[B-drops] what the dropped directions carried is beyond every "
f"decoder: of the {len(twins)} pairs of days whose codes lie closer "
f"than 0.1 together, {stamps[p0]} and {stamps[q0]} differ most in wind "
f"speed, {wind[p0]:.1f} against {wind[q0]:.1f} meters per second, and "
f"the decoder returns {rebuilt[0]:.2f} meters per second for both",
wind_gap > 3.0 and abs(rebuilt[0] - rebuilt[1]) < 0.05)
windy = np.where(wind > np.median(wind), 1.0, -1.0)
def ls_accuracy(F):
"""Accuracy of a linear classifier learned from F by least squares."""
G = np.hstack([F, np.ones((len(F), 1))])
wls = np.linalg.lstsq(G, windy, rcond=None)[0]
return float(np.mean(np.sign(G @ wls) == windy))
acc_raw, acc_code = ls_accuracy(X), ls_accuracy(Z)
acc_calm = float(np.mean(windy < 0))
check(f"[B-drops] a day counts as windy when its mean wind speed lies above "
f"the median of the {m} values, which holds for "
f"{int(np.sum(windy > 0))} of them; a linear classifier learned by "
f"least squares has an accuracy of {100 * acc_raw:.0f} percent from "
f"all {d} scaled measurements and {100 * acc_code:.0f} percent from "
f"the {DPRIME} code numbers, against {100 * acc_calm:.0f} percent for "
f"calling every day calm",
acc_code < acc_raw - 0.2 and acc_code < acc_calm + 0.05)
centers = [X[windy == c].mean(axis=0) for c in (-1.0, 1.0)]
scatter = sum(np.mean(windy == c)
* ((X[windy == c] - centers[s]).T @ (X[windy == c] - centers[s]))
/ np.sum(windy == c) for s, c in enumerate((-1.0, 1.0)))
w_lda = np.linalg.solve(scatter, centers[1] - centers[0])
acc_lda = ls_accuracy((X @ w_lda).reshape(-1, 1))
check(f"[B-drops] one new feature chosen from the labels instead of from "
f"the sample covariance matrix, the linear discriminant analysis "
f"direction, reaches {100 * acc_lda:.0f} percent: a code learned to "
f"reconstruct well is not a code learned to predict well",
acc_lda > acc_code + 0.2)
month = np.array([int(day[5:7]) for day in stamps])
coldhalf = (month <= 3) | (month >= 10)
Xcold = X[coldhalf]
center_cold = Xcold.mean(axis=0)
evals_c, evecs_c = np.linalg.eigh(
(Xcold - center_cold).T @ (Xcold - center_cold) / len(Xcold))
Wcold = evecs_c[:, np.argsort(evals_c)[::-1][:DPRIME]].T
Ain = Xcold - center_cold
err_in = float(np.mean(np.sum((Ain - (Ain @ Wcold.T) @ Wcold) ** 2, axis=1)))
Aout = X[~coldhalf] - center_cold
err_out = float(np.mean(np.sum((Aout - (Aout @ Wcold.T) @ Wcold) ** 2,
axis=1)))
Awarm = X[~coldhalf]
err_year = float(np.mean(np.sum((Awarm - (Awarm @ W.T) @ W) ** 2, axis=1)))
check(f"[B-drops] an encoder is fixed once learned: the PCA encoder of the "
f"{int(np.sum(coldhalf))} days of January to March and of October to "
f"December leaves {err_in:.2f} on those days and {err_out:.2f} on the "
f"other {int(np.sum(~coldhalf))}, against {err_year:.2f} for the "
f"encoder of the whole year on the same days",
err_out > 3 * err_year)
[B-drops] what the code of two numbers drops [ok] [B-drops] the code of 2 numbers does not keep the eight measurements equally well: of the unit sample variance of each, it keeps 96 percent of the mean temperature and 3 percent of the wind speed [ok] [B-drops] what the dropped directions carried is beyond every decoder: of the 80 pairs of days whose codes lie closer than 0.1 together, 2024-02-14 and 2024-11-22 differ most in wind speed, 1.0 against 4.5 meters per second, and the decoder returns 1.76 meters per second for both [ok] [B-drops] a day counts as windy when its mean wind speed lies above the median of the 366 values, which holds for 130 of them; a linear classifier learned by least squares has an accuracy of 98 percent from all 8 scaled measurements and 66 percent from the 2 code numbers, against 64 percent for calling every day calm [ok] [B-drops] one new feature chosen from the labels instead of from the sample covariance matrix, the linear discriminant analysis direction, reaches 98 percent: a code learned to reconstruct well is not a code learned to predict well [ok] [B-drops] an encoder is fixed once learned: the PCA encoder of the 183 days of January to March and of October to December leaves 1.93 on those days and 10.87 on the other 183, against 2.48 for the encoder of the whole year on the same days
Preview: the two-number code of 21 days, and the reconstruction error against the code length.
print("[B-plot] CSVs and the matplotlib preview")
with open(OUT_DIR / "encoder_code.csv", "w") as fh:
fh.write("date,z1,z2,warm\n")
for day, z, wflag in zip(stamps, Z, warm):
fh.write(f"{day},{z[0]:.4f},{z[1]:.4f},{int(wflag)}\n")
keep = range(0, m, 18) # every 18th day, 21 of 366
with open(OUT_DIR / "encoder_code_plot.csv", "w") as fh:
fh.write("z1,z2,warm,month\n") # numeric columns only
for i in keep:
fh.write(f"{Z[i, 0]:.4f},{Z[i, 1]:.4f},{int(warm[i])},"
f"{int(stamps[i][5:7])}\n")
check(f"[B-plot] the thinned file holds every 18th day, {len(list(keep))} of "
f"the {m}", len(list(keep)) == 21)
with open(OUT_DIR / "encoder_errors.csv", "w") as fh:
fh.write("dprime,pca,random\n")
for k, (p, r) in enumerate(zip(errs_pca, errs_rnd), start=1):
fh.write(f"{k},{p:.6f},{r:.6f}\n")
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(9.4, 3.9))
idx = list(keep)
cold = [i for i in idx if not warm[i]]
hot = [i for i in idx if warm[i]]
ax1.plot(Z[cold, 0], Z[cold, 1], ls="none", marker="s", mfc="none",
color="black", ms=5, label="cold day (open squares)")
ax1.plot(Z[hot, 0], Z[hot, 1], ls="none", marker="o", color="black", ms=4,
label="warm day (filled circles)")
ax1.set_xlabel("first code number z1")
ax1.set_ylabel("second code number z2")
ax1.set_title("the two-number code of 21 of the 366 days")
ax1.legend(frameon=False, fontsize=8)
ks = np.arange(1, d + 1)
ax2.plot(ks, errs_pca, "k-", label="PCA encoder (solid curve)")
ax2.plot(ks, errs_rnd, "k--",
label="random orthonormal encoder (dashed curve)")
ax2.set_xlabel("code length d'")
ax2.set_ylabel("average squared reconstruction error")
ax2.set_title("a longer code leaves less to rebuild")
ax2.legend(frameon=False, fontsize=8)
fig.tight_layout()
fig.savefig(OUT_DIR / "encoder.png", dpi=110)
n_ok = sum(ok for _, ok in report)
print(f"\n{n_ok}/{len(report)} checks pass")
if n_ok != len(report):
raise SystemExit(1)
[B-plot] CSVs and the matplotlib preview [ok] [B-plot] the thinned file holds every 18th day, 21 of the 366 21/21 checks pass

B-plot writes when the script runs