{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3",
   "language": "python"
  },
  "language_info": {
   "name": "python"
  },
  "colab": {
   "name": "ppca.ipynb"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# probabilistic principal component analysis (probabilistic PCA) \u2014 Python demo\n\nNumerical companion to the entry [probabilistic principal component analysis (probabilistic PCA)](https://dictionaryofml.org/terms/ppca.html) of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org/): it recomputes what the entry states and prints one line per check.\n\nNumerical companion to the glossary entry 'ppca' (probabilistic principal component analysis). The GeoSphere Austria weather station Krems (station id 3805) records eight measurements per day: lowest, highest and mean air temperature, precipitation, sunshine duration, relative humidity, air pressure and wind speed. A value of -1 for precipitation marks a trace of rain too small to record and is set to 0.\n\nRequires NumPy and Matplotlib only, and uses fixed seeds, so the printed numbers reproduce exactly. Generated from [`pythondemos/ppca.py`](https://dictionaryofml.org/terms/ppca.py); CC BY 4.0.\n\nBefore running a downloaded copy, scan it: `uvx bandit -q -t B102,B307,B602,B605 ppca.py` (B102,B307,B602,B605 are the exec, eval and shell checks of [bandit](https://bandit.readthedocs.io/)) reports nothing: the script runs no shell command and evaluates no generated code."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "# Notebook shim: the script resolves output paths relative to __file__,\n# which a notebook kernel does not define; everything lands in the\n# working directory instead.\nimport os\n__file__ = os.path.join(os.getcwd(), \"ppca.py\")\nos.makedirs(\"pythondemos\", exist_ok=True)"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "\"\"\"Probabilistic PCA on the days at Krems an der Donau: a latent Gaussian\ncode, a matrix and isotropic Gaussian noise give a density over the\nmeasurements of a day, whose maximum likelihood estimate is read off the\neigenvalues of the sample covariance matrix.\n\nPurpose\n-------\nNumerical companion to the glossary entry 'ppca' (probabilistic principal\ncomponent analysis).  The GeoSphere Austria weather station Krems\n(station id 3805) records eight measurements per day: lowest, highest and\nmean air temperature, precipitation, sunshine duration, relative\nhumidity, air pressure and wind speed.  A value of -1 for precipitation\nmarks a trace of rain too small to record and is set to 0.\n\nProbabilistic PCA models the feature vector of a day as x = W z + mu + e,\nwith a latent code z drawn from a zero-mean unit-covariance Gaussian, a\nd x d' matrix W, and isotropic Gaussian noise e of variance sigma^2.  The\nfeature vector is then Gaussian with mean mu and covariance\nC = W W^T + sigma^2 I, so every day gets a probability density value.\n\nThe demo checks the entry's claims: (1) the covariance C of the model\nhas the retained eigenvalues of the sample covariance matrix along the\nprincipal directions and sigma^2 in every direction orthogonal to them;\n(2) the maximum likelihood estimate has a closed form in those\neigenvalues, with sigma^2 the average of the discarded ones, and it beats\nrandom parameters and a grid over sigma^2; (3) the EM algorithm is a\nfixed-point iteration whose fixed point is that closed form; (4) the loss\n-log p is the one PCA minimizes in the limit of vanishing noise, since\n2 sigma^2 times its average over the training set tends to the minimum\nreconstruction error of PCA; (5) the posterior mean of the latent code is\na shrunk version of the PCA coordinates and becomes the orthogonal\nprojection as sigma^2 tends to zero; (6) the density scores a day that did\nnot belong to the training set, and the least probable day of 2024 is the\nday of the September flood; (7) the same density fills in a missing\nreading by a conditional mean; and (8) both assumptions of the model fail\non this\nrecord: the days nearly satisfy a linear relation that the single noise\nvariance cannot hold, the learned subspace depends on the units of the\nmeasurements, factor analysis with one variance per measurement scores\nthe held-out days higher, and far more days of 2024 fall in the tail of\nthe density than a Gaussian allows.\n\nTwo views of the same data, as in the 'pca' demo.  The picture uses the\ntwo temperatures in degrees Celsius, centered but not rescaled, so the\ndensity can be drawn in the units of the measurement.  The numbers quoted\nfor the eight measurements use all of them, each scaled to zero sample\nmean and unit variance on the 2023 days because temperatures,\nprecipitation and pressure carry different units.\n\nSelf-contained (numpy and matplotlib only), fixed seeds; the GeoSphere\ndownload falls back to the committed CSV.\n\nBlocks\n------\n[B-data]    The eight measurements for the 365 days of 2023 and the 366\n            days of 2024; the two temperatures centered for the picture\n            and all eight scaled for the numbers.\n[B-model]   The model covariance C = W W^T + sigma^2 I: its eigenvalues,\n            and the sample covariance matrix of a large number of draws.\n[B-mle]     The closed-form maximum likelihood estimate: sigma^2 is the\n            average of the discarded eigenvalues, and the estimate beats\n            2000 random parameter sets and a grid over sigma^2.\n[B-em]      EM as a fixed-point iteration: it reaches the log-likelihood\n            of the closed form, and the closed form is a fixed point of\n            the EM map.\n[B-erm]     Maximum likelihood as empirical risk minimization: the loss\n            is -log p, and 2 sigma^2 times its average over the training\n            set tends to the minimum reconstruction error of PCA as the\n            noise variance tends to zero.\n[B-limit]   The posterior mean of the latent code shrinks the PCA\n            coordinates by a known factor and becomes the orthogonal\n            projection as sigma^2 tends to zero.\n[B-density] A density for every day: the log-density of the 366 days of\n            2024 under the model learned on the 365 days of 2023, and the\n            least probable day of 2024.\n[B-missing] The conditional mean under the same density fills in a\n            deleted highest-temperature reading.\n[B-costs]   Where the two assumptions fail: the isotropic noise against\n            the near-exact linear relation among the three temperatures\n            and against a rescaled measurement, factor analysis as the\n            alternative, and the Gaussian tail against the number of\n            days of 2024 that fall into it.\n[B-plot]    Preview: the days of 2023 in the two temperatures with draws\n            from the model and a level curve of its density, and the\n            log-density of each day of 2024 over the year.\n\nOutputs\n-------\nppca_weather.csv     : date and the eight measurements, 2023 and 2024\nppca_points_plot.csv : x1, x2 -- the centered temperatures of the 20 of\n                       the 365 days of 2023 the figure draws (every 19th)\nppca_draws_plot.csv  : x1, x2 -- 20 draws from the learned model\nppca_ellipse.csv     : x1, x2 -- the level curve of the density one\n                       standard deviation out\nppca_axis.csv        : x1, x2 -- the principal direction as a segment\nppca_logdens.csv     : day of the year and the log-density of that day of\n                       2024 under the learned model\nppca.png             : matplotlib preview (checking only)\n\"\"\"\n\n# /// script\n# requires-python = \">=3.10\"\n# dependencies = [\"matplotlib==3.10.7\", \"numpy==2.3.4\"]\n# ///\n\nimport json\nimport math\nimport urllib.request\nfrom pathlib import Path\n\nimport numpy as np\nimport matplotlib\n\nmatplotlib.use(\"Agg\")\nimport matplotlib.pyplot as plt\n\nOUT_DIR = Path(__file__).parent\n\nreport = []\n\n\ndef check(name, ok):\n    report.append((name, bool(ok)))\n    print(f\"  [{'ok' if ok else 'FAIL'}] {name}\")"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-data]** The eight measurements for the 365 days of 2023 and the 366 days of 2024; the two temperatures centered for the picture and all eight scaled for the numbers."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-data] the eight measurements at Krems for 2023 and 2024\")\nPARAMS = [\"tlmin\", \"tlmax\", \"tl_mittel\", \"rr\", \"so_h\", \"rf_mittel\",\n          \"p_mittel\", \"vv_mittel\"]\nURL = (\"https://dataset.api.hub.geosphere.at/v1/station/historical/\"\n       f\"klima-v2-1d?parameters={','.join(PARAMS)}&station_ids=3805\"\n       \"&start=2023-01-01&end=2024-12-31\")\nCSV = OUT_DIR / \"ppca_weather.csv\"\ntry:\n    with urllib.request.urlopen(URL, timeout=180) as resp:\n        payload = json.load(resp)\n    params = payload[\"features\"][0][\"properties\"][\"parameters\"]\n    stamps = [t[:10] for t in payload[\"timestamps\"]]\n    M = np.stack([np.array(params[p][\"data\"], dtype=float) for p in PARAMS], 1)\n    M[:, 3] = np.maximum(M[:, 3], 0.0)         # -1 marks a trace of rain\n    with open(CSV, \"w\") as fh:\n        fh.write(\"date,\" + \",\".join(PARAMS) + \"\\n\")\n        for day, row in zip(stamps, M):\n            fh.write(day + \",\" + \",\".join(f\"{v:g}\" for v in row) + \"\\n\")\n    print(f\"    downloaded {len(stamps)} days\")\nexcept OSError as err:\n    print(f\"    download failed ({err}); reading the committed CSV\")\n    rows = CSV.read_text().strip().splitlines()[1:]\n    stamps = [r.split(\",\")[0] for r in rows]\n    M = np.array([[float(v) for v in r.split(\",\")[1:]] for r in rows])\n\nis23 = np.array([s.startswith(\"2023\") for s in stamps])\ncheck(\"[B-data] 365 days of 2023 and 366 days of 2024, eight measurements \"\n      \"each\", M.shape == (731, 8) and is23.sum() == 365)\ncheck(\"[B-data] no measurement is missing\", not np.isnan(M).any())\ncheck(\"[B-data] the record matches the archive (1 February 2024: lowest \"\n      \"-3.8, highest 10.4 degrees)\",\n      stamps[396] == \"2024-02-01\" and np.allclose(M[396, :2], [-3.8, 10.4]))\n\nT = M[is23, :2] - M[is23, :2].mean(axis=0)     # two temperatures, centered\nmu8, sd8 = M[is23].mean(axis=0), M[is23].std(axis=0)\nX23 = (M[is23] - mu8) / sd8                    # all eight, scaled on 2023\nX24 = (M[~is23] - mu8) / sd8\nm23, m24, d = len(X23), len(X24), M.shape[1]\ncheck(\"[B-data] the two temperatures of the picture are centered\",\n      np.allclose(T.mean(axis=0), 0))\n\n\ndef principal(A):\n    \"\"\"Eigenvalues and eigenvectors of the sample covariance matrix of the\n    centered rows of A, in decreasing order of eigenvalue.\"\"\"\n    B = A - A.mean(axis=0)\n    vals, vecs = np.linalg.eigh(B.T @ B / len(B))\n    order = np.argsort(vals)[::-1]\n    return vals[order], vecs[:, order]\n\n\ndef ppca_mle(A, dp):\n    \"\"\"Closed-form maximum likelihood estimate of the model parameters for\n    a code of dp numbers: the sample mean, W = U (L - sigma^2 I)^(1/2) and\n    sigma^2 the average of the discarded eigenvalues.\"\"\"\n    vals, vecs = principal(A)\n    sig2 = float(vals[dp:].mean())\n    W = vecs[:, :dp] * np.sqrt(np.maximum(vals[:dp] - sig2, 0.0))\n    return A.mean(axis=0), W, sig2, vals, vecs\n\n\ndef log_density_cov(A, mean, C):\n    \"\"\"Log of the density of a multivariate normal distribution with the\n    given mean vector and covariance matrix at every row of A.\"\"\"\n    _, logdet = np.linalg.slogdet(C)\n    B = A - mean\n    quad = np.einsum(\"ij,jk,ik->i\", B, np.linalg.inv(C), B)\n    return -0.5 * (A.shape[1] * np.log(2 * np.pi) + logdet + quad)\n\n\ndef log_density(A, mean, W, sig2):\n    \"\"\"Log of the density of the model at every row of A.\"\"\"\n    dim = A.shape[1]\n    return log_density_cov(A, mean, W @ W.T + sig2 * np.eye(dim))"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-model]** The model covariance C = W W^T + sigma^2 I: its eigenvalues, and the sample covariance matrix of a large number of draws."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-model] the covariance of the model and the draws it produces\")\nDCODE = 3\nmean8, W8, sig28, lam8, U8 = ppca_mle(X23, DCODE)\nC8 = W8 @ W8.T + sig28 * np.eye(d)\ncvals, _ = np.linalg.eigh(C8)\ncvals = np.sort(cvals)[::-1]\nprint(f\"    eigenvalues of the sample covariance matrix: \"\n      f\"{np.array2string(lam8, precision=3)}\")\nprint(f\"    eigenvalues of C for d' = {DCODE}: \"\n      f\"{np.array2string(cvals, precision=3)}\")\ncheck(f\"[B-model] C keeps the {DCODE} largest eigenvalues of the sample \"\n      f\"covariance matrix exactly\", np.allclose(cvals[:DCODE], lam8[:DCODE]))\ncheck(f\"[B-model] C replaces the other {d - DCODE} by sigma^2 = \"\n      f\"{sig28:.4f}, their average\", np.allclose(cvals[DCODE:], sig28))\ncheck(\"[B-model] C is the sum of a matrix of rank d' and a multiple of \"\n      \"the identity matrix\",\n      np.linalg.matrix_rank(W8 @ W8.T, tol=1e-9) == DCODE)\nrng = np.random.default_rng(0)\nZ = rng.standard_normal((400000, DCODE))\nE = np.sqrt(sig28) * rng.standard_normal((400000, d))\ndraws8 = Z @ W8.T + mean8 + E\nChat = (draws8 - draws8.mean(axis=0)).T @ (draws8 - draws8.mean(axis=0)) / len(draws8)\ncheck(f\"[B-model] 400000 draws W z + mu + e have sample covariance matrix \"\n      f\"within {np.abs(Chat - C8).max():.4f} of C\",\n      np.abs(Chat - C8).max() < 0.02)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-mle]** The closed-form maximum likelihood estimate: sigma^2 is the average of the discarded eigenvalues, and the estimate beats 2000 random parameter sets and a grid over sigma^2."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-mle] the closed-form maximum likelihood estimate\")\nll_mle = float(log_density(X23, mean8, W8, sig28).sum())\ncheck(f\"[B-mle] sigma^2 = {sig28:.4f} is the average of the \"\n      f\"{d - DCODE} discarded eigenvalues\", np.isclose(sig28, lam8[DCODE:].mean()))\nworse = 0\nfor _ in range(2000):\n    Wr = rng.standard_normal((d, DCODE)) * np.sqrt(lam8[:DCODE]).mean()\n    s2r = float(np.exp(rng.uniform(np.log(1e-3), np.log(2.0))))\n    if log_density(X23, mean8, Wr, s2r).sum() < ll_mle:\n        worse += 1\ncheck(f\"[B-mle] the estimate beats all 2000 random parameter sets \"\n      f\"(log-likelihood {ll_mle:.1f})\", worse == 2000)\ngrid = np.exp(np.linspace(np.log(1e-3), np.log(2.0), 400))\nbest = max(grid, key=lambda s2: log_density(\n    X23, mean8, U8[:, :DCODE] * np.sqrt(np.maximum(lam8[:DCODE] - s2, 0.0)),\n    s2).sum())\ncheck(f\"[B-mle] a grid of 400 values of sigma^2 is best at {best:.4f}, \"\n      f\"next to the closed form {sig28:.4f}\", abs(best - sig28) < 0.02)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-em]** EM as a fixed-point iteration: it reaches the log-likelihood of the closed form, and the closed form is a fixed point of the EM map."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-em] EM as a fixed-point iteration on (W, sigma^2)\")\n\n\ndef em_step(A, W, sig2):\n    \"\"\"One EM step for the model parameters, the map T of the entry.\"\"\"\n    dim, dp = W.shape\n    B = A - A.mean(axis=0)\n    S = B.T @ B / len(B)\n    Mi = np.linalg.inv(W.T @ W + sig2 * np.eye(dp))\n    SW = S @ W\n    Wn = SW @ np.linalg.inv(sig2 * np.eye(dp) + Mi @ W.T @ SW)\n    sig2n = float(np.trace(S - SW @ Mi @ Wn.T) / dim)\n    return Wn, sig2n\n\n\nW_em = rng.standard_normal((d, DCODE))\ns2_em = 1.0\nlls = []\nfor _ in range(4000):\n    W_em, s2_em = em_step(X23, W_em, s2_em)\n    lls.append(float(log_density(X23, mean8, W_em, s2_em).sum()))\ncheck(\"[B-em] every EM step increases the log-likelihood\",\n      np.all(np.diff(lls) > -1e-6))\ncheck(f\"[B-em] EM reaches the log-likelihood of the closed form \"\n      f\"({lls[-1]:.3f} against {ll_mle:.3f})\", abs(lls[-1] - ll_mle) < 1e-3)\ncheck(f\"[B-em] EM recovers sigma^2 = {s2_em:.4f}\", abs(s2_em - sig28) < 1e-4)\nC_em = W_em @ W_em.T + s2_em * np.eye(d)\ncheck(f\"[B-em] the covariance it delivers agrees with C to \"\n      f\"{np.abs(C_em - C8).max():.2e}, so the rotation of W is the only \"\n      f\"part left open\", np.abs(C_em - C8).max() < 1e-4)\nW_fix, s2_fix = em_step(X23, W8, sig28)\ncheck(f\"[B-em] the closed form is a fixed point of T: one more step moves \"\n      f\"it by {max(np.abs(W_fix - W8).max(), abs(s2_fix - sig28)):.2e}\",\n      np.abs(W_fix - W8).max() < 1e-10 and abs(s2_fix - sig28) < 1e-10)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-erm]** Maximum likelihood as empirical risk minimization: the loss is -log p, and 2 sigma^2 times its average over the training set tends to the minimum reconstruction error of PCA as the noise variance tends to zero."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-erm] maximum likelihood as ERM, and the loss as sigma^2 vanishes\")\n\n\ndef mean_loss(A, mean, W, sig2):\n    \"\"\"The average of the loss -log p over the rows of A, the empirical\n    risk of the negative log density loss.\"\"\"\n    return float(-log_density(A, mean, W, sig2).mean())\n\n\ndef profile_loss(sig2, vals, dim, dp):\n    \"\"\"Twice that average written in the eigenvalues of the sample\n    covariance matrix, Eq. (17) of Tipping and Bishop (1999).\"\"\"\n    return (dim * np.log(2 * np.pi) + float(np.log(vals[:dp]).sum())\n            + float(vals[dp:].sum()) / sig2\n            + (dim - dp) * np.log(sig2) + dp)\n\n\nerr_pca = float(lam8[DCODE:].sum())\nproj = U8[:, :DCODE] @ U8[:, :DCODE].T\ncen = X23 - mean8\ndist2 = ((cen - cen @ proj) ** 2).sum(axis=1)\ncolnorm = np.linalg.norm(W8, axis=0)\ncheck(f\"[B-erm] the columns of W are orthogonal with lengths \"\n      f\"{np.array2string(colnorm, precision=2)} = sqrt(lambda_j - sigma^2), \"\n      f\"so W has no orthonormal columns\",\n      np.allclose(W8.T @ W8, np.diag(lam8[:DCODE] - sig28))\n      and not np.allclose(colnorm, 1.0))\ncheck(f\"[B-erm] the squared error loss of PCA averages {dist2.mean():.4f} \"\n      f\"over the {m23} days, the sum of the {d - DCODE} discarded \"\n      f\"eigenvalues\", np.isclose(dist2.mean(), err_pca))\nRATIO_TOL = {1e-2: 3e-2, 1e-4: 3e-3, 1e-6: 1e-4}\ngaps = []\nfor s2 in (1e-2, 1e-4, 1e-6):\n    W_s = U8[:, :DCODE] * np.sqrt(lam8[:DCODE] - s2)\n    risk = mean_loss(X23, mean8, W_s, s2)\n    ratio = 2 * s2 * risk / err_pca\n    gaps.append(abs(ratio - 1.0))\n    check(f\"[B-erm] at sigma^2 = {s2:g}, 2 sigma^2 times the empirical risk \"\n          f\"is {2 * s2 * risk:.4f} against the {err_pca:.4f} of PCA, a ratio \"\n          f\"of {ratio:.4f}\", abs(ratio - 1.0) < RATIO_TOL[s2])\n    prof = profile_loss(s2, lam8, d, DCODE)\n    check(f\"[B-erm] and twice that risk agrees with Eq. (17) of Tipping and \"\n          f\"Bishop (1999) to {abs(prof - 2 * risk) / abs(2 * risk):.1e} \"\n          f\"relative\", abs(prof - 2 * risk) < 1e-8 * abs(2 * risk))\n    pointwise = -2 * s2 * log_density(X23, mean8, W_s, s2)\n    check(f\"[B-erm] day by day it stays within \"\n          f\"{np.abs(pointwise - dist2).max():.1e} of the squared distance of \"\n          f\"the day from the retained subspace\",\n          np.abs(pointwise - dist2).max() < 40 * s2 * abs(np.log(s2)))\ncheck(f\"[B-erm] the ratio approaches one as sigma^2 falls: the gaps are \"\n      f\"{', '.join(f'{g:.1e}' for g in gaps)}\",\n      gaps[0] > gaps[1] > gaps[2])"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-limit]** The posterior mean of the latent code shrinks the PCA coordinates by a known factor and becomes the orthogonal projection as sigma^2 tends to zero."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-limit] the posterior mean of the code against the PCA coordinates\")\n\n\ndef posterior_mean(A, mean, W, sig2):\n    \"\"\"The mean of the posterior distribution of the latent code.\"\"\"\n    dp = W.shape[1]\n    Mi = np.linalg.inv(W.T @ W + sig2 * np.eye(dp))\n    return (A - mean) @ W @ Mi\n\n\npca_coord = (X23 - mean8) @ U8[:, :DCODE]      # PCA coordinates\nz_post = posterior_mean(X23, mean8, W8, sig28)\nratio = np.sqrt(np.maximum(lam8[:DCODE] - sig28, 0.0)) / lam8[:DCODE]\ncheck(\"[B-limit] the posterior mean is the PCA coordinate scaled by \"\n      f\"sqrt(lambda_j - sigma^2)/lambda_j = \"\n      f\"{np.array2string(ratio, precision=4)}, so it is pulled toward the \"\n      \"origin\", np.allclose(z_post, pca_coord * ratio))\ncheck(\"[B-limit] every factor is below one\",\n      np.all(ratio < 1.0) and np.all(ratio > 0.0))\nTINY = 1e-12\nW0 = U8[:, :DCODE] * np.sqrt(lam8[:DCODE] - TINY)\nz0 = posterior_mean(X23, mean8, W0, TINY)\nproj_ppca = z0 @ W0.T + mean8\nproj_pca = pca_coord @ U8[:, :DCODE].T + mean8\ncheck(f\"[B-limit] at sigma^2 = {TINY:g} the reconstruction W z + mu agrees \"\n      f\"with the PCA projection to \"\n      f\"{np.abs(proj_ppca - proj_pca).max():.2e}\",\n      np.abs(proj_ppca - proj_pca).max() < 1e-8)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-density]** A density for every day: the log-density of the 366 days of 2024 under the model learned on the 365 days of 2023, and the least probable day of 2024."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-density] the log-density of the 366 days of 2024\")\ndays24 = [s for s in stamps if s.startswith(\"2024\")]\nld24 = log_density(X24, mean8, W8, sig28)\nld23 = log_density(X23, mean8, W8, sig28)\norder = np.argsort(ld24)\nlow, med = order[0], float(np.median(ld24))\nprint(f\"    median {med:.2f}, lowest {ld24[low]:.2f} ({days24[low]}), \"\n      f\"highest {ld24[order[-1]]:.2f}\")\ncheck(f\"[B-density] the average log-density of the 2024 days is \"\n      f\"{ld24.mean():.3f}, below the {ld23.mean():.3f} of the 2023 days \"\n      f\"the model was learned from\", ld24.mean() < ld23.mean())\ncheck(f\"[B-density] the least probable day of 2024 is {days24[low]}, day \"\n      f\"{low + 1} of the year, at a log-density of {ld24[low]:.1f} against \"\n      f\"{ld24[order[1]]:.1f} for the next lowest and {med:.1f} for the \"\n      f\"median day\",\n      days24[low] == \"2024-09-14\" and ld24[low] < ld24[order[1]] - 300)\nrain_z = (M[~is23][low, 3] - mu8[3]) / sd8[3]\ncheck(f\"[B-density] that day recorded {M[~is23][low, 3]:.1f} mm of \"\n      f\"precipitation, {rain_z:.1f} sample standard deviations above the \"\n      f\"2023 average of {mu8[3]:.1f} mm\", rain_z > 20.0)\ntail = [days24[j] for j in np.flatnonzero(ld24 < -40)]\ncheck(f\"[B-density] {len(tail)} of the {m24} days fall below a log-density \"\n      f\"of -40: {', '.join(tail)}; the lowest of the remaining days is \"\n      f\"{ld24[ld24 >= -40].min():.1f}\",\n      0 < len(tail) < 12)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-missing]** The conditional mean under the same density fills in a deleted highest-temperature reading."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-missing] filling in a deleted highest-temperature reading\")\nobs = [j for j in range(d) if j != 1]\nCoo = C8[np.ix_(obs, obs)]\nCmo = C8[1, obs]\nfilled = (mean8[1] + (X24[:, obs] - mean8[obs]) @ np.linalg.solve(Coo, Cmo))\nfilled_deg = filled * sd8[1] + mu8[1]\ntruth_deg = M[~is23, 1]\nrmse = float(np.sqrt(np.mean((filled_deg - truth_deg) ** 2)))\nrmse_mean = float(np.sqrt(np.mean((mu8[1] - truth_deg) ** 2)))\ncheck(f\"[B-missing] the conditional mean fills in the deleted reading with \"\n      f\"a root mean squared error of {rmse:.2f} degrees over the {m24} \"\n      f\"days, against {rmse_mean:.2f} degrees for the average of 2023\",\n      rmse < rmse_mean)\nfeb1 = days24.index(\"2024-02-01\")\ncheck(f\"[B-missing] on 1 February 2024 it reports \"\n      f\"{filled_deg[feb1]:.1f} degrees against the recorded \"\n      f\"{truth_deg[feb1]:.1f}\",\n      abs(filled_deg[feb1] - truth_deg[feb1]) < 6.0)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-costs]** Where the two assumptions fail: the isotropic noise against the near-exact linear relation among the three temperatures and against a rescaled measurement, factor analysis as the alternative, and the Gaussian tail against the number of days of 2024 that fall into it."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-costs] what the two assumptions of the model cost on this record\")\ngap = M[is23, 2] - 0.5 * (M[is23, 0] + M[is23, 1])\ncheck(f\"[B-costs] the mean air temperature of a day is the average of its \"\n      f\"lowest and its highest to within {np.abs(gap).max():.2f} degrees on \"\n      f\"every one of the {m23} days of 2023, so the eight measurements \"\n      f\"nearly satisfy a linear relation\", np.abs(gap).max() < 0.1)\nprint(f\"    the {d - DCODE} discarded eigenvalues: \"\n      f\"{np.array2string(lam8[DCODE:], precision=6)}\")\ncheck(f\"[B-costs] the days spread by {np.sqrt(lam8[-1]):.4f} along the last \"\n      f\"direction, while the isotropic noise expects \"\n      f\"{np.sqrt(sig28):.2f}, a variance larger by a factor of \"\n      f\"{sig28 / lam8[-1]:.0f}\", sig28 / lam8[-1] > 1e4)\ncheck(f\"[B-costs] the variance the model gives one standardized measurement \"\n      f\"is off by up to {np.abs(np.diag(C8) - 1).max():.2f} from the 1 the \"\n      f\"scaling set\", np.abs(np.diag(C8) - 1).max() > 0.25)\nRESCALE = 10.0\nscale = np.ones(d)\nscale[3] = RESCALE                             # precipitation in other units\n_, _, _, _, U_sc = ppca_mle(X23 * scale, DCODE)\ncosines = np.linalg.svd(U8[:, :DCODE].T @ U_sc[:, :DCODE], compute_uv=False)\nangle = float(np.degrees(np.arccos(np.clip(cosines.min(), -1.0, 1.0))))\ncheck(f\"[B-costs] multiplying the precipitation reading by {RESCALE:g} turns \"\n      f\"the retained subspace by {angle:.0f} degrees, measured as the \"\n      f\"largest angle between the two subspaces\", angle > 45.0)\n\n\ndef factor_analysis(A, dp, iters=4000, seed=1):\n    \"\"\"Maximum likelihood factor analysis by EM: the same model with the\n    noise covariance a diagonal matrix, one variance per measurement.\"\"\"\n    n, dim = A.shape\n    mean = A.mean(axis=0)\n    B = A - mean\n    S = B.T @ B / n\n    W = np.random.default_rng(seed).standard_normal((dim, dp)) * 0.3\n    psi = np.diag(S).copy()\n    for _ in range(iters):\n        Mi = np.linalg.inv(np.eye(dp) + (W.T / psi) @ W)\n        beta = Mi @ (W.T / psi)\n        W = (S @ beta.T) @ np.linalg.inv(Mi + beta @ S @ beta.T)\n        psi = np.maximum(np.diag(S - W @ beta @ S), 1e-8)\n    return mean, W, psi\n\n\nmean_fa, W_fa, psi_fa = factor_analysis(X23, DCODE)\nC_fa = W_fa @ W_fa.T + np.diag(psi_fa)\nld24_fa = log_density_cov(X24, mean_fa, C_fa)\nprint(f\"    the noise variances of factor analysis: \"\n      f\"{np.array2string(psi_fa, precision=5)}\")\ncheck(f\"[B-costs] factor analysis raises the median log-density of the \"\n      f\"{m24} days of 2024 from {med:.2f} to {np.median(ld24_fa):.2f}, with \"\n      f\"the model parameters learned from the same {m23} days of 2023\",\n      np.median(ld24_fa) > med)\ncheck(f\"[B-costs] it gives the three temperatures a noise variance of at \"\n      f\"most {psi_fa[:3].max():.1e}, the near-exact relation the single \"\n      f\"sigma^2 cannot hold\", psi_fa[:3].max() < 1e-3)\n\n\ndef chi2_upper(x, dof=8):\n    \"\"\"Probability that a chi-squared RV with an even number dof of degrees\n    of freedom exceeds x.\"\"\"\n    terms = [(x / 2) ** i / float(math.factorial(i))\n             for i in range(dof // 2)]\n    return float(np.exp(-x / 2) * sum(terms))\n\n\ndef chi2_percentile(p, dof=8):\n    \"\"\"The value a chi-squared RV with dof degrees of freedom stays below\n    with probability p, by bisection on the upper tail.\"\"\"\n    lo, hi = 0.0, 1.0e4\n    for _ in range(200):\n        mid = 0.5 * (lo + hi)\n        if chi2_upper(mid, dof) > 1.0 - p:\n            lo = mid\n        else:\n            hi = mid\n    return 0.5 * (lo + hi)\n\n\nquad24 = np.einsum(\"ij,jk,ik->i\", X24 - mean8, np.linalg.inv(C8),\n                   X24 - mean8)\nquad23 = np.einsum(\"ij,jk,ik->i\", X23 - mean8, np.linalg.inv(C8),\n                   X23 - mean8)\nq99 = chi2_percentile(0.99)\ncheck(f\"[B-costs] the 99th percentile of a chi-squared RV with {d} degrees \"\n      f\"of freedom is {q99:.1f}, so {0.01 * m24:.1f} of the {m24} days of \"\n      f\"2024 would exceed it under the model and {int((quad24 > q99).sum())} \"\n      f\"do, as do {int((quad23 > q99).sum())} of the {m23} days of 2023\",\n      (quad24 > q99).sum() > 4 * 0.01 * m24)\ncheck(f\"[B-costs] on 14 September 2024 that quadratic form is \"\n      f\"{quad24[low]:.1f}, which the chi-squared RV exceeds with \"\n      f\"probability {chi2_upper(quad24[low]):.1e}\",\n      chi2_upper(quad24[low]) < 1e-100)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-plot]** Preview: the days of 2023 in the two temperatures with draws from the model and a level curve of its density, and the log-density of each day of 2024 over the year."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "print(\"[B-plot] the two-temperature model and the preview\")\nmean2, W2, sig22, lam2, U2 = ppca_mle(T, 1)\nC2 = W2 @ W2.T + sig22 * np.eye(2)\ncheck(f\"[B-plot] with one of the two directions kept, sigma^2 = {sig22:.2f} is \"\n      f\"the one discarded eigenvalue {lam2[1]:.2f} squared degrees\",\n      np.isclose(sig22, lam2[1]))\ncheck(f\"[B-plot] the learned direction is the principal direction of the \"\n      f\"two temperatures (eigenvalues {lam2[0]:.1f} and {lam2[1]:.1f})\",\n      np.isclose(abs(W2[:, 0] @ U2[:, 0]) / np.linalg.norm(W2[:, 0]), 1.0))\ndraws2 = (rng.standard_normal((20, 1)) @ W2.T + mean2\n          + np.sqrt(sig22) * rng.standard_normal((20, 2)))\nangles = np.linspace(0, 2 * np.pi, 241)\nL = np.linalg.cholesky(C2)\nellipse = (L @ np.stack([np.cos(angles), np.sin(angles)])).T + mean2\nTHIN = 19\npts = T[::THIN]\ncheck(f\"[B-plot] the figure draws every {THIN}th of the {m23} days of \"\n      f\"2023, which is {len(pts)} of them\", len(pts) == 20)\nseg = np.stack([-2 * np.sqrt(lam2[0]) * U2[:, 0],\n                2 * np.sqrt(lam2[0]) * U2[:, 0]]) + mean2\n\nwith open(OUT_DIR / \"ppca_points_plot.csv\", \"w\") as fh:\n    fh.write(\"x1,x2\\n\")\n    for a, b in pts:\n        fh.write(f\"{a:.4f},{b:.4f}\\n\")\nwith open(OUT_DIR / \"ppca_draws_plot.csv\", \"w\") as fh:\n    fh.write(\"x1,x2\\n\")\n    for a, b in draws2:\n        fh.write(f\"{a:.4f},{b:.4f}\\n\")\nwith open(OUT_DIR / \"ppca_ellipse.csv\", \"w\") as fh:\n    fh.write(\"x1,x2\\n\")\n    for a, b in ellipse:\n        fh.write(f\"{a:.4f},{b:.4f}\\n\")\nwith open(OUT_DIR / \"ppca_axis.csv\", \"w\") as fh:\n    fh.write(\"x1,x2\\n\")\n    for a, b in seg:\n        fh.write(f\"{a:.4f},{b:.4f}\\n\")\nwith open(OUT_DIR / \"ppca_logdens.csv\", \"w\") as fh:\n    fh.write(\"day,logdens\\n\")\n    for i, s in enumerate(ld24):\n        fh.write(f\"{i + 1},{s:.4f}\\n\")\n\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(9.6, 4.0))\nax1.plot(pts[:, 0], pts[:, 1], \"o\", ms=4, color=\"black\",\n         label=\"20 of the 365 days of 2023 (filled circles)\")\nax1.plot(draws2[:, 0], draws2[:, 1], \"^\", ms=5, mfc=\"none\", color=\"black\",\n         label=\"20 draws from the learned model (open triangles)\")\nax1.plot(ellipse[:, 0], ellipse[:, 1], \"-\", color=\"black\",\n         label=\"level curve of the density (solid curve)\")\nax1.plot(seg[:, 0], seg[:, 1], \"--\", color=\"black\",\n         label=\"principal direction (dashed line)\")\nax1.set_xlabel(\"lowest temperature of the day in degrees, centered\")\nax1.set_ylabel(\"highest temperature in degrees, centered\")\nax1.set_title(\"a density around the principal direction\")\nax1.legend(frameon=False, fontsize=7)\nax2.plot(np.arange(1, m24 + 1), ld24, \"-\", lw=0.8, color=\"black\",\n         label=\"log-density of each day of 2024 (solid curve)\")\nax2.plot([low + 1], [ld24[low]], \"s\", ms=5, color=\"black\",\n         label=f\"{days24[low]} (filled square)\")\nax2.set_xlabel(\"day of the year 2024\")\nax2.set_ylabel(\"log-density of the day\")\nax2.set_title(\"every day gets a density value\")\nax2.legend(frameon=False, fontsize=7, loc=\"lower left\")\nfig.tight_layout()\nfig.savefig(OUT_DIR / \"ppca.png\", dpi=110)\n\nn_ok = sum(ok for _, ok in report)\nprint(f\"\\n{n_ok}/{len(report)} checks pass\")\nif n_ok != len(report):\n    raise SystemExit(1)"
  }
 ]
}