"""Encoders on a year of days at Krems an der Donau and on an excerpt of
the Universal Declaration of Human Rights: a map from the raw features of
a data point to a shorter code, judged by how well a decoder rebuilds the
data point from that code.

Purpose
-------
Numerical companion to the glossary entry 'encoder'.  The GeoSphere
Austria weather station Krems an der Donau (station id 3805) records
eight measurements per day.  The principal component analysis (PCA)
encoder is the matrix whose rows are the eigenvectors of the sample
covariance matrix belonging to its largest eigenvalues; it maps the eight
scaled measurements of a day to a code of d' numbers, and the decoder,
its transpose, maps the code back to eight numbers.

The demo recomputes what the entry claims.  (1) Encoder followed by
decoder returns approximately the day it started from: with a code of two
numbers the eight measurements of 1 January 2024 come back to within
0.80 on the scaled axes, and the reconstruction error averaged over the
366 days is 2.91 of the 8.00 total variance.  (2) The reconstruction
error falls with the code length d' and equals the sum of the eigenvalues
the code drops, down to zero at d' = 8.  (3) The encoder matters, not
only the code length: a fixed random orthonormal encoder leaves a larger
reconstruction error at every code length below 8.  (4) The token-to-
vector step of a language model is an encoder of the same shape -- the
one-hot vector of a token times the embedding matrix is the token's row
of that matrix -- and here it loses nothing, since distinct rows let the
nearest row recover every token.  (5) What the code of two numbers drops
is beyond every decoder and can matter for a later learning task: it
keeps 96 percent of the sample variance of the mean temperature and 3
percent of the wind speed, two days with almost the same code differ by
3.5 meters per second in wind speed while the decoder returns one value
for both, a linear classifier learned by least squares calls windy days
with an accuracy of 66 percent from the code against 98 percent from
the eight measurements, and one new feature chosen from the labels, the
linear discriminant analysis direction, reaches 98 percent as well.

Deterministic: the weather data are a fixed archive year, PCA is an
eigenvalue decomposition, and the random encoder and the embedding matrix
are drawn from fixed seeds.  Self-contained: numpy and matplotlib only
(stdlib urllib for the download, which falls back to the committed CSV).

Blocks
------
[B-data]    Download the 366 days of 2024 with eight measurements each and
            scale every measurement to zero sample mean and unit sample
            variance, so that the eight carry comparable weight.
[B-encoder] The PCA encoder for a code of two numbers: the rows of the
            matrix are eigenvectors of the sample covariance matrix, the
            code of a day is the matrix times its scaled measurements, and
            the decoder is the transpose. Encoder then decoder returns the
            day approximately; the entry's named day is reported.
[B-length]  The reconstruction error against the code length d' = 1 to 8:
            it equals the sum of the dropped eigenvalues and vanishes at
            d' = 8, already at d' = 7 because the mean temperature is the
            midpoint of the lowest and the highest, and a fixed random
            orthonormal encoder leaves more at every code length below 8.
[B-token]   The token-to-vector encoder of a language model on an excerpt
            of the Universal Declaration of Human Rights: a one-hot vector
            of length |V| times the embedding matrix selects a row, and the
            nearest row recovers every token, so this encoder loses nothing.
[B-drops]   What the code of two numbers drops: the kept fraction of each
            measurement, a pair of days with almost the same code and very
            different wind speeds, the accuracy of a linear classifier for
            windy days from the code against the eight measurements and
            against one linear discriminant analysis feature, and the error
            an encoder learned on the cold half leaves on the warm half.
[B-plot]    Preview: the two-number code of 21 days, and the reconstruction
            error against the code length.

Outputs
-------
encoder_weather.csv    : date and the eight measurements (committed copy)
encoder_code.csv       : date, z1, z2, warm -- the code of all 366 days
encoder_code_plot.csv  : z1, z2, warm, month -- every 18th of those days,
                         the 21 dots of the figure
encoder_errors.csv     : dprime, pca, random -- error against code length
encoder.png            : matplotlib preview (checking only)
"""

# /// script
# requires-python = ">=3.10"
# dependencies = ["matplotlib==3.10.7", "numpy==2.3.4"]
# ///

import json
import urllib.request
from pathlib import Path

import numpy as np
import matplotlib

matplotlib.use("Agg")
import matplotlib.pyplot as plt

OUT_DIR = Path(__file__).parent

report = []


def check(name, ok):
    report.append((name, bool(ok)))
    print(f"  [{'ok' if ok else 'FAIL'}] {name}")


# ---- [B-data] the eight daily measurements at Krems for 2024
print("[B-data] eight daily measurements at Krems an der Donau, 2024")
PARAMS = ["tlmin", "tlmax", "tl_mittel", "rr", "so_h", "rf_mittel",
          "p_mittel", "vv_mittel"]
URL = ("https://dataset.api.hub.geosphere.at/v1/station/historical/"
       f"klima-v2-1d?parameters={','.join(PARAMS)}&station_ids=3805"
       "&start=2024-01-01&end=2024-12-31")
CSV = OUT_DIR / "encoder_weather.csv"
try:
    with urllib.request.urlopen(URL, timeout=120) as resp:
        payload = json.load(resp)
    params = payload["features"][0]["properties"]["parameters"]
    stamps = [t[:10] for t in payload["timestamps"]]
    M = np.stack([np.array(params[p]["data"], dtype=float) for p in PARAMS], 1)
    M[:, 3] = np.maximum(M[:, 3], 0.0)        # -1 marks a day without rain
    with open(CSV, "w") as fh:
        fh.write("date," + ",".join(PARAMS) + "\n")
        for day, row in zip(stamps, M):
            fh.write(day + "," + ",".join(f"{v:g}" for v in row) + "\n")
    print(f"    downloaded {len(stamps)} days")
except OSError as err:
    print(f"    download failed ({err}); reading the committed CSV")
    rows = CSV.read_text().strip().splitlines()[1:]
    stamps = [r.split(",")[0] for r in rows]
    M = np.array([[float(v) for v in r.split(",")[1:]] for r in rows])

check("[B-data] 366 days with eight measurements each", M.shape == (366, 8))
check("[B-data] no day is missing a measurement", np.all(np.isfinite(M)))
X = (M - M.mean(axis=0)) / M.std(axis=0)
m, d = X.shape
check(f"[B-data] after scaling, every measurement has sample mean 0 and "
      f"sample variance 1, so the {d} together carry a total variance of "
      f"{float(np.sum(np.var(X, axis=0))):.2f}",
      abs(float(np.sum(np.var(X, axis=0))) - d) < 1e-9)

# ---- [B-encoder] the PCA encoder for a code of two numbers
print("[B-encoder] the PCA encoder: eigenvectors of the sample covariance")
C = X.T @ X / m                                # sample covariance matrix
evals, evecs = np.linalg.eigh(C)
order = np.argsort(evals)[::-1]
evals, evecs = evals[order], evecs[:, order]
check(f"[B-encoder] the eight eigenvalues of the sample covariance matrix "
      f"sum to the total variance {float(np.sum(evals)):.2f}; the two "
      f"largest are {evals[0]:.2f} and {evals[1]:.2f}",
      abs(float(np.sum(evals)) - d) < 1e-9)

DPRIME = 2
W = evecs[:, :DPRIME].T                        # encoder, d' by d
Z = X @ W.T                                    # the code of every day
Xhat = Z @ W                                   # decoder: the transpose
check(f"[B-encoder] the encoder has shape ({DPRIME}, {d}) and its rows are "
      f"orthonormal", W.shape == (DPRIME, d)
      and np.allclose(W @ W.T, np.eye(DPRIME)))
err2 = float(np.mean(np.sum((X - Xhat) ** 2, axis=1)))
check(f"[B-encoder] averaged over the {m} days, the squared distance between "
      f"a day and its reconstruction is {err2:.2f} of the total variance "
      f"{d:.2f}, so a code of {DPRIME} numbers keeps "
      f"{100 * (1 - err2 / d):.0f} percent of it", 0 < err2 < d)
i0 = stamps.index("2024-01-01")
gap0 = np.abs(X[i0] - Xhat[i0])
check(f"[B-encoder] encoder then decoder returns the day approximately: for "
      f"1 January 2024 the code is ({Z[i0, 0]:.2f}, {Z[i0, 1]:.2f}) and the "
      f"eight reconstructed measurements differ from the scaled originals "
      f"by at most {gap0.max():.2f}", gap0.max() < 1.0)
warm = X[:, 1] > np.median(X[:, 1])            # maximum temperature above median
check(f"[B-encoder] the first code number separates the {int(np.sum(warm))} "
      f"days whose maximum temperature is above the median from the "
      f"{int(np.sum(~warm))} others: its mean is {Z[warm, 0].mean():.2f} on "
      f"the warm days and {Z[~warm, 0].mean():.2f} on the cold ones",
      abs(Z[warm, 0].mean() - Z[~warm, 0].mean()) > 1.0)

# ---- [B-length] reconstruction error against the code length
print("[B-length] reconstruction error against the code length")
rng = np.random.default_rng(2024)
Q = np.linalg.qr(rng.standard_normal((d, d)))[0]     # random orthonormal rows
errs_pca, errs_rnd, dropped = [], [], []
for k in range(1, d + 1):
    Wk = evecs[:, :k].T
    errs_pca.append(float(np.mean(np.sum((X - (X @ Wk.T) @ Wk) ** 2, axis=1))))
    Rk = Q[:, :k].T
    errs_rnd.append(float(np.mean(np.sum((X - (X @ Rk.T) @ Rk) ** 2, axis=1))))
    dropped.append(float(np.sum(evals[k:])))
check(f"[B-length] the error of the PCA encoder equals the sum of the "
      f"eigenvalues the code drops at every code length (largest difference "
      f"{max(abs(a - b) for a, b in zip(errs_pca, dropped)):.2e})",
      np.allclose(errs_pca, dropped))
check(f"[B-length] the error falls with the code length, from "
      f"{errs_pca[0]:.2f} at one number to {errs_pca[1]:.2f} at two and "
      f"{errs_pca[3]:.2f} at four, and vanishes at {d}: {errs_pca[-1]:.1e}",
      np.all(np.diff(errs_pca) < 0) and errs_pca[-1] < 1e-12)
check(f"[B-length] the smallest eigenvalue is {evals[-1]:.1e}, so one of the "
      f"eight measurements adds nothing the other seven do not already "
      f"carry: a code of seven numbers leaves {errs_pca[6]:.1e}",
      evals[-1] < 1e-3)
check(f"[B-length] the random orthonormal encoder leaves more error at every "
      f"code length below {d} ({errs_rnd[1]:.2f} against {errs_pca[1]:.2f} at "
      f"two numbers), so the encoder matters and not only the code length",
      all(r > p + 1e-9 for r, p in zip(errs_rnd[:-1], errs_pca[:-1])))

# ---- [B-token] the token-to-vector encoder of a language model
print("[B-token] the token-to-vector encoder on a UDHR excerpt")
EXCERPT = ("All human beings are born free and equal in dignity and rights. "
           "They are endowed with reason and conscience and should act "
           "towards one another in a spirit of brotherhood. Everyone has "
           "the right to life, liberty and security of person.")
tokens = EXCERPT.lower().replace(",", "").replace(".", "").split()
vocab = sorted(set(tokens))
nv, dtok = len(vocab), 4
check(f"[B-token] the excerpt has {len(tokens)} tokens over a vocabulary of "
      f"{nv} distinct ones, and each is encoded as a vector of {dtok} "
      f"numbers", nv < len(tokens) and dtok < nv)
rng_tok = np.random.default_rng(7)
E = rng_tok.standard_normal((nv, dtok))        # the embedding matrix
v = vocab.index("dignity")
onehot = np.zeros(nv)
onehot[v] = 1.0
check(f"[B-token] the one-hot vector of a token times the embedding matrix "
      f"is that token's row: for token {v} the two agree to "
      f"{float(np.max(np.abs(onehot @ E - E[v]))):.1e}",
      np.allclose(onehot @ E, E[v]))
codes = E[[vocab.index(t) for t in tokens]]
recovered = [int(np.argmin(np.sum((E - c) ** 2, axis=1))) for c in codes]
check(f"[B-token] the nearest row recovers the token for all {len(tokens)} "
      f"tokens, so this encoder shortens a vector of length {nv} to one of "
      f"length {dtok} and still loses nothing",
      recovered == [vocab.index(t) for t in tokens])

# ---- [B-drops] what the code of two numbers drops, and whether it matters
print("[B-drops] what the code of two numbers drops")
kept = 1.0 - np.mean((X - Xhat) ** 2, axis=0)
check(f"[B-drops] the code of {DPRIME} numbers does not keep the eight "
      f"measurements equally well: of the unit sample variance of each, it "
      f"keeps {100 * kept[2]:.0f} percent of the mean temperature and "
      f"{100 * kept[7]:.0f} percent of the wind speed",
      kept[2] > 0.95 and kept[7] < 0.05)
wind = M[:, 7]
code_dist = np.sqrt(((Z[:, None, :] - Z[None, :, :]) ** 2).sum(axis=2))
np.fill_diagonal(code_dist, np.inf)
twins = [(abs(wind[p] - wind[q]), p, q) for p in range(m)
         for q in range(p + 1, m) if code_dist[p, q] < 0.1]
twins.sort(reverse=True)
wind_gap, p0, q0 = twins[0]
rebuilt = Xhat[[p0, q0], 7] * wind.std() + wind.mean()
check(f"[B-drops] what the dropped directions carried is beyond every "
      f"decoder: of the {len(twins)} pairs of days whose codes lie closer "
      f"than 0.1 together, {stamps[p0]} and {stamps[q0]} differ most in wind "
      f"speed, {wind[p0]:.1f} against {wind[q0]:.1f} meters per second, and "
      f"the decoder returns {rebuilt[0]:.2f} meters per second for both",
      wind_gap > 3.0 and abs(rebuilt[0] - rebuilt[1]) < 0.05)

windy = np.where(wind > np.median(wind), 1.0, -1.0)


def ls_accuracy(F):
    """Accuracy of a linear classifier learned from F by least squares."""
    G = np.hstack([F, np.ones((len(F), 1))])
    wls = np.linalg.lstsq(G, windy, rcond=None)[0]
    return float(np.mean(np.sign(G @ wls) == windy))


acc_raw, acc_code = ls_accuracy(X), ls_accuracy(Z)
acc_calm = float(np.mean(windy < 0))
check(f"[B-drops] a day counts as windy when its mean wind speed lies above "
      f"the median of the {m} values, which holds for "
      f"{int(np.sum(windy > 0))} of them; a linear classifier learned by "
      f"least squares has an accuracy of {100 * acc_raw:.0f} percent from "
      f"all {d} scaled measurements and {100 * acc_code:.0f} percent from "
      f"the {DPRIME} code numbers, against {100 * acc_calm:.0f} percent for "
      f"calling every day calm",
      acc_code < acc_raw - 0.2 and acc_code < acc_calm + 0.05)
centers = [X[windy == c].mean(axis=0) for c in (-1.0, 1.0)]
scatter = sum(np.mean(windy == c)
              * ((X[windy == c] - centers[s]).T @ (X[windy == c] - centers[s]))
              / np.sum(windy == c) for s, c in enumerate((-1.0, 1.0)))
w_lda = np.linalg.solve(scatter, centers[1] - centers[0])
acc_lda = ls_accuracy((X @ w_lda).reshape(-1, 1))
check(f"[B-drops] one new feature chosen from the labels instead of from "
      f"the sample covariance matrix, the linear discriminant analysis "
      f"direction, reaches {100 * acc_lda:.0f} percent: a code learned to "
      f"reconstruct well is not a code learned to predict well",
      acc_lda > acc_code + 0.2)

month = np.array([int(day[5:7]) for day in stamps])
coldhalf = (month <= 3) | (month >= 10)
Xcold = X[coldhalf]
center_cold = Xcold.mean(axis=0)
evals_c, evecs_c = np.linalg.eigh(
    (Xcold - center_cold).T @ (Xcold - center_cold) / len(Xcold))
Wcold = evecs_c[:, np.argsort(evals_c)[::-1][:DPRIME]].T
Ain = Xcold - center_cold
err_in = float(np.mean(np.sum((Ain - (Ain @ Wcold.T) @ Wcold) ** 2, axis=1)))
Aout = X[~coldhalf] - center_cold
err_out = float(np.mean(np.sum((Aout - (Aout @ Wcold.T) @ Wcold) ** 2,
                               axis=1)))
Awarm = X[~coldhalf]
err_year = float(np.mean(np.sum((Awarm - (Awarm @ W.T) @ W) ** 2, axis=1)))
check(f"[B-drops] an encoder is fixed once learned: the PCA encoder of the "
      f"{int(np.sum(coldhalf))} days of January to March and of October to "
      f"December leaves {err_in:.2f} on those days and {err_out:.2f} on the "
      f"other {int(np.sum(~coldhalf))}, against {err_year:.2f} for the "
      f"encoder of the whole year on the same days",
      err_out > 3 * err_year)

# ---- [B-plot] CSVs and the preview
print("[B-plot] CSVs and the matplotlib preview")
with open(OUT_DIR / "encoder_code.csv", "w") as fh:
    fh.write("date,z1,z2,warm\n")
    for day, z, wflag in zip(stamps, Z, warm):
        fh.write(f"{day},{z[0]:.4f},{z[1]:.4f},{int(wflag)}\n")
keep = range(0, m, 18)                          # every 18th day, 21 of 366
with open(OUT_DIR / "encoder_code_plot.csv", "w") as fh:
    fh.write("z1,z2,warm,month\n")              # numeric columns only
    for i in keep:
        fh.write(f"{Z[i, 0]:.4f},{Z[i, 1]:.4f},{int(warm[i])},"
                 f"{int(stamps[i][5:7])}\n")
check(f"[B-plot] the thinned file holds every 18th day, {len(list(keep))} of "
      f"the {m}", len(list(keep)) == 21)
with open(OUT_DIR / "encoder_errors.csv", "w") as fh:
    fh.write("dprime,pca,random\n")
    for k, (p, r) in enumerate(zip(errs_pca, errs_rnd), start=1):
        fh.write(f"{k},{p:.6f},{r:.6f}\n")

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(9.4, 3.9))
idx = list(keep)
cold = [i for i in idx if not warm[i]]
hot = [i for i in idx if warm[i]]
ax1.plot(Z[cold, 0], Z[cold, 1], ls="none", marker="s", mfc="none",
         color="black", ms=5, label="cold day (open squares)")
ax1.plot(Z[hot, 0], Z[hot, 1], ls="none", marker="o", color="black", ms=4,
         label="warm day (filled circles)")
ax1.set_xlabel("first code number z1")
ax1.set_ylabel("second code number z2")
ax1.set_title("the two-number code of 21 of the 366 days")
ax1.legend(frameon=False, fontsize=8)
ks = np.arange(1, d + 1)
ax2.plot(ks, errs_pca, "k-", label="PCA encoder (solid curve)")
ax2.plot(ks, errs_rnd, "k--",
         label="random orthonormal encoder (dashed curve)")
ax2.set_xlabel("code length d'")
ax2.set_ylabel("average squared reconstruction error")
ax2.set_title("a longer code leaves less to rebuild")
ax2.legend(frameon=False, fontsize=8)
fig.tight_layout()
fig.savefig(OUT_DIR / "encoder.png", dpi=110)

n_ok = sum(ok for _, ok in report)
print(f"\n{n_ok}/{len(report)} checks pass")
if n_ok != len(report):
    raise SystemExit(1)
