"""Soft clustering of the days of one year at Krems an der Donau: the
degree-of-belonging vector of a day as a point of the probability simplex.

Purpose
-------
Numerical companion to the glossary entry 'softclustering'.  The 366
days of 2024 at Krems are data points with the feature vector (morning
minimum temperature, maximum daytime temperature).  A hard clustering
(k-means) assigns every day to one of two clusters, the cold and the
warm days.  A soft clustering assigns every day a degree of belonging
to each cluster: a vector with nonnegative entries that sum to one, a
point of the probability simplex.  A hard clustering is the special
case where that vector is a vertex of the simplex.

The demo checks the entry's claims: the posterior probabilities of a
two-component Gaussian mixture model (GMM), learned by EM, are such
degree-of-belonging vectors and the negative log-likelihood never
increases along the EM iterations; a hard clustering is recovered by
the largest degree of belonging; with the covariance matrices fixed at
sigma^2 I and the cluster probabilities at 1/2, the degrees of
belonging turn into 0 or 1 as sigma^2 shrinks and then agree with the
nearest-centroid assignment of k-means; fuzzy c-means delivers degrees
of belonging without a probabilistic model, its alternating updates
never increase its objective, and it reduces to k-means as the
fuzzifier tends to 1.

Deterministic: Lloyd's algorithm starts from k-means++ seeds drawn with
a fixed seed; EM and fuzzy c-means start from the k-means centroids.
Self-contained: numpy + matplotlib only; the GeoSphere download falls
back to the committed CSV.

Blocks
------
[B-data]   Download the daily minimum and maximum temperature of 2024
           at Krems (GeoSphere station 3805) or read the committed
           copy; 366 days, no missing value.
[B-hard]   k-means with k = 2 (Lloyd's algorithm, best of 20 k-means++
           restarts): the two centroids, the cold and the warm days;
           the degree-of-belonging vector of every day is a vertex of
           the simplex.
[B-gmm]    EM for a two-component GMM with full covariance matrices,
           started at the k-means centroids: the degrees of belonging
           are the posterior probabilities, they sum to one for every
           day, the negative log-likelihood never increases, and the
           days whose largest degree of belonging is below 0.8 are
           counted; the clustering by the largest degree of belonging
           is compared with k-means.
[B-limit]  The constrained GMM with cluster probabilities 1/2 and
           covariance matrices sigma^2 I, means fixed at the k-means
           centroids: the number of days with a degree of belonging
           between 0.2 and 0.8 for sigma^2 = 25, 4, 1, and the
           agreement of the largest degree of belonging with the
           nearest centroid.
[B-fcm]    Fuzzy c-means with fuzzifier 2 (and 1.05, 10): the
           alternating updates never increase the objective, the
           degrees of belonging sum to one, the ambiguous days are
           counted, and the fuzzifier 1.05 recovers the k-means
           centroids and clustering error.
[B-plot]   Every 18th day for the figure (21 of 366 days) with the
           cluster of its largest GMM degree of belonging and whether
           that value reaches 0.8; the k-means centroids and their
           bisector; the curve of equal GMM degrees of belonging; and
           the degree of belonging to the cold cluster along the line
           through the two centroids under k-means, the GMM, the
           constrained GMM with sigma^2 = 4, and fuzzy c-means.

Outputs
-------
softclustering_weather.csv     : date, tmin, tmax (committed copy).
softclustering_points_plot.csv : the 21 drawn days, cluster, confident.
softclustering_centroids.csv   : the two k-means centroids.
softclustering_bisector.csv    : two points of the k-means decision
                                 boundary.
softclustering_contour.csv     : the curve of equal GMM posteriors.
softclustering_segment.csv     : s and the four degrees of belonging
                                 to the cold cluster along the line.
softclustering.png             : matplotlib preview (checking only).
"""

# /// script
# requires-python = ">=3.10"
# dependencies = ["matplotlib==3.10.7", "numpy==2.3.4"]
# ///

import json
import urllib.request
from pathlib import Path

import numpy as np
import matplotlib

matplotlib.use("Agg")
import matplotlib.pyplot as plt

OUT_DIR = Path(__file__).parent

report = []


def check(name, ok):
    report.append((name, bool(ok)))
    print(f"  [{'ok' if ok else 'FAIL'}] {name}")


# ---- [B-data] daily minimum and maximum temperature at Krems, 2024
print("[B-data] daily minimum and maximum temperature at Krems, 2024")
URL = ("https://dataset.api.hub.geosphere.at/v1/station/historical/"
       "klima-v2-1d?parameters=tlmin,tlmax&station_ids=3805"
       "&start=2024-01-01&end=2024-12-31")
CSV = OUT_DIR / "softclustering_weather.csv"
try:
    with urllib.request.urlopen(URL, timeout=120) as resp:
        payload = json.load(resp)
    params = payload["features"][0]["properties"]["parameters"]
    records = [(stamp[:10], lo, hi)
               for stamp, lo, hi in zip(payload["timestamps"],
                                        params["tlmin"]["data"],
                                        params["tlmax"]["data"])]
    with open(CSV, "w") as fh:
        fh.write("date,tmin,tmax\n")
        for day, lo, hi in records:
            fh.write(f"{day},{lo},{hi}\n")
    print(f"    downloaded {len(records)} days")
except OSError as err:
    print(f"    download failed ({err}); reading the committed CSV")
    rows = CSV.read_text().strip().splitlines()[1:]
    records = [(d, float(a), float(b))
               for d, a, b in (r.split(",") for r in rows)]

check("[B-data] 366 days for 2024", len(records) == 366)
check("[B-data] no day is missing a temperature",
      all(lo is not None and hi is not None for _, lo, hi in records))
X = np.array([[lo, hi] for _, lo, hi in records], dtype=float)
m, d = X.shape
K = 2


def sq_euclid(x, w):
    return np.sum((x - w) ** 2, axis=-1)


def sq_dist_matrix(X, W):
    """m x k matrix of squared Euclidean distances to the centroids W."""
    return np.stack([sq_euclid(X, w) for w in W], axis=1)


# ---- [B-hard] k-means: a hard clustering, DOB vectors at the vertices
print("[B-hard] k-means with k = 2")


def lloyd(X, W0, max_iter=500):
    W = np.array(W0, dtype=float)
    c = np.argmin(sq_dist_matrix(X, W), axis=1)
    for _ in range(max_iter):
        W = np.array([X[c == j].mean(axis=0) if np.any(c == j) else W[j]
                      for j in range(len(W))])
        c_new = np.argmin(sq_dist_matrix(X, W), axis=1)
        if np.array_equal(c_new, c):
            break
        c = c_new
    return W, c, float(np.mean(np.min(sq_dist_matrix(X, W), axis=1)))


def kmeanspp_seeds(X, k, rng):
    W = [X[rng.integers(m)]]
    for _ in range(1, k):
        d2 = np.min(sq_dist_matrix(X, np.array(W)), axis=1)
        W.append(X[rng.choice(m, p=d2 / d2.sum())])
    return np.array(W)


rng = np.random.default_rng(2024)
best = None
for _ in range(20):
    W, c, err = lloyd(X, kmeanspp_seeds(X, K, rng))
    if best is None or err < best[2]:
        best = (W, c, err)
W_km, c_km, err_km = best
cold, warm = sorted(range(K), key=lambda j: W_km[j, 1])
if cold != 0:  # cluster 1 is the cold one throughout
    W_km = W_km[[cold, warm]]
    c_km = np.where(c_km == cold, 0, 1)
    cold, warm = 0, 1
U_km = np.eye(K)[c_km]  # one-hot rows: the vertices of the simplex
check(f"[B-hard] centroids ({W_km[0, 0]:.1f}, {W_km[0, 1]:.1f}) with "
      f"{int(np.sum(c_km == 0))} cold days and ({W_km[1, 0]:.1f}, "
      f"{W_km[1, 1]:.1f}) with {int(np.sum(c_km == 1))} warm days; "
      f"clustering error {err_km:.1f}",
      W_km[0, 1] < 12 < W_km[1, 1])
check("[B-hard] the degree-of-belonging vector of every day is a vertex "
      "of the simplex: entries in {0, 1} that sum to one",
      np.all(np.isin(U_km, (0.0, 1.0))) and np.allclose(U_km.sum(axis=1), 1))


# ---- [B-gmm] EM for a two-component GMM; the DOB is the posterior
print("[B-gmm] EM for a two-component GMM with full covariance matrices")


def mvn_pdf(X, mu, C):
    Ci = np.linalg.inv(C)
    diff = X - mu
    q = np.einsum("ij,jk,ik->i", diff, Ci, diff)
    return np.exp(-0.5 * q) / np.sqrt((2 * np.pi) ** d * np.linalg.det(C))


def gmm_posterior(X, p, mu, C):
    dens = np.stack([p[j] * mvn_pdf(X, mu[j], C[j]) for j in range(K)],
                    axis=1)
    return dens / dens.sum(axis=1, keepdims=True), -np.mean(
        np.log(dens.sum(axis=1)))


def em_gmm(X, mu0, max_iter=1000, tol=1e-12):
    p = np.full(K, 1.0 / K)
    mu = np.array(mu0, dtype=float)
    C = np.array([np.cov(X.T)] * K)
    nll = []
    for _ in range(max_iter):
        U, value = gmm_posterior(X, p, mu, C)  # E-step
        nll.append(value)
        Nk = U.sum(axis=0)  # M-step
        p = Nk / m
        mu = (U.T @ X) / Nk[:, None]
        for j in range(K):
            diff = X - mu[j]
            C[j] = (U[:, j, None] * diff).T @ diff / Nk[j]
        if len(nll) > 1 and nll[-2] - nll[-1] < tol:
            break
    U, value = gmm_posterior(X, p, mu, C)
    nll.append(value)
    return p, mu, C, U, nll


p_g, mu_g, C_g, U_g, nll = em_gmm(X, W_km)
check(f"[B-gmm] EM stopped after {len(nll) - 1} iterations; cluster "
      f"probabilities ({p_g[0]:.2f}, {p_g[1]:.2f}), means "
      f"({mu_g[0, 0]:.1f}, {mu_g[0, 1]:.1f}) and ({mu_g[1, 0]:.1f}, "
      f"{mu_g[1, 1]:.1f})", mu_g[0, 1] < mu_g[1, 1])
check("[B-gmm] the negative log-likelihood never increases along the "
      f"EM iterations ({nll[0]:.3f} -> {nll[-1]:.3f})",
      all(b <= a + 1e-9 for a, b in zip(nll, nll[1:])))
check("[B-gmm] every day's degrees of belonging are nonnegative and sum "
      "to one (a point of the simplex)",
      np.all(U_g >= 0) and np.allclose(U_g.sum(axis=1), 1))
n_amb_g = int(np.sum(U_g.max(axis=1) < 0.8))
check(f"[B-gmm] {n_amb_g} of the {m} days have no degree of belonging "
      f"of 0.8 or more", 0 < n_amb_g < m / 4)
c_g = np.argmax(U_g, axis=1)
agree_g = float(np.mean(c_g == c_km))
check(f"[B-gmm] the cluster of the largest degree of belonging agrees "
      f"with k-means on {100 * agree_g:.1f}% of the days "
      f"({int(np.sum(c_g == c_km))} of {m})", agree_g > 0.85)
months = np.array([int(day[5:7]) for day, _, _ in records])
amb_months = np.bincount(months[U_g.max(axis=1) < 0.8], minlength=13)
check("[B-gmm] no day of January, February, November or December is "
      f"ambiguous; ambiguous days per month: {amb_months[1:].tolist()}",
      amb_months[[1, 2, 11, 12]].sum() == 0)


# ---- [B-limit] constrained GMM: sigma^2 I and probabilities 1/2
print("[B-limit] the constrained GMM at the k-means centroids as sigma^2 "
      "shrinks")


def constrained_dob(X, W, sigma2):
    """Posterior under equal cluster probabilities and covariance
    sigma^2 I with means W; computed in the log domain."""
    L = -sq_dist_matrix(X, W) / (2 * sigma2)
    L -= L.max(axis=1, keepdims=True)
    U = np.exp(L)
    return U / U.sum(axis=1, keepdims=True)


between = {}
for sigma2 in (25.0, 4.0, 1.0):
    U_s = constrained_dob(X, W_km, sigma2)
    between[sigma2] = int(np.sum((U_s[:, 0] > 0.2) & (U_s[:, 0] < 0.8)))
    check(f"[B-limit] sigma^2 = {sigma2:g}: {between[sigma2]} days with a "
          f"degree of belonging between 0.2 and 0.8; the largest degree "
          f"of belonging names the nearest centroid for every day",
          np.array_equal(np.argmax(U_s, axis=1), c_km))
check("[B-limit] the count of days between 0.2 and 0.8 falls with "
      "sigma^2 and reaches zero",
      between[25.0] > between[4.0] > between[1.0] == 0)
U_tiny = constrained_dob(X, W_km, 1e-4)
check("[B-limit] sigma^2 = 0.0001: every degree of belonging is 0 or 1 "
      "to within 1e-6 and equals the k-means vertex",
      np.allclose(U_tiny, U_km, atol=1e-6))


# ---- [B-fcm] fuzzy c-means
print("[B-fcm] fuzzy c-means")


def fcm_dob(X, W, phi):
    D2 = np.maximum(sq_dist_matrix(X, W), 1e-12)
    U = D2 ** (-1.0 / (phi - 1.0))
    return U / U.sum(axis=1, keepdims=True)


def fcm_objective(X, W, U, phi):
    return float(np.mean(np.sum(U ** phi * sq_dist_matrix(X, W), axis=1)))


def fcm(X, W0, phi, max_iter=1000, tol=1e-10):
    """Alternating updates: the degrees of belonging for fixed centroids,
    then the centroids for fixed degrees of belonging.  Returns the
    centroids, the degrees of belonging, and the objective after every
    half step."""
    W = np.array(W0, dtype=float)
    values = []
    for _ in range(max_iter):
        U = fcm_dob(X, W, phi)
        values.append(fcm_objective(X, W, U, phi))
        W_new = (U ** phi).T @ X / (U ** phi).sum(axis=0)[:, None]
        values.append(fcm_objective(X, W_new, U, phi))
        if np.allclose(W_new, W, atol=tol):
            W = W_new
            break
        W = W_new
    return W, fcm_dob(X, W, phi), values


fcm_runs = {}
for phi in (2.0, 1.05, 10.0):
    fcm_runs[phi] = fcm(X, W_km + 0.5, phi)
W_f, U_f, vals_f = fcm_runs[2.0]
check(f"[B-fcm] fuzzifier 2: {len(vals_f) // 2} iterations, the objective "
      f"never increases ({vals_f[0]:.2f} -> {vals_f[-1]:.2f}); centroids "
      f"({W_f[0, 0]:.1f}, {W_f[0, 1]:.1f}) and ({W_f[1, 0]:.1f}, "
      f"{W_f[1, 1]:.1f})",
      all(b <= a + 1e-9 for a, b in zip(vals_f, vals_f[1:])))
check("[B-fcm] fuzzifier 2: every day's degrees of belonging are "
      "nonnegative and sum to one",
      np.all(U_f >= 0) and np.allclose(U_f.sum(axis=1), 1))
n_amb_f = int(np.sum(U_f.max(axis=1) < 0.8))
check(f"[B-fcm] fuzzifier 2: {n_amb_f} days have no degree of belonging "
      f"of 0.8 or more (GMM: {n_amb_g})", 0 < n_amb_f < m / 4)
check("[B-fcm] fuzzifier 2: the largest degree of belonging names the "
      "nearest fuzzy c-means centroid for every day",
      np.array_equal(np.argmax(U_f, axis=1),
                     np.argmin(sq_dist_matrix(X, W_f), axis=1)))
W_1, U_1, vals_1 = fcm_runs[1.05]
n_sharp = int(np.sum(U_1.max(axis=1) > 0.99))
check(f"[B-fcm] fuzzifier 1.05: centroids within 0.05 of the k-means "
      f"centroids, objective {vals_1[-1]:.1f} within 0.1 of the k-means "
      f"clustering error {err_km:.1f}, and {n_sharp} of the {m} days have "
      f"a largest degree of belonging above 0.99",
      np.allclose(W_1, W_km, atol=0.05) and abs(vals_1[-1] - err_km) < 0.1
      and n_sharp >= m - 10)
W_10, U_10, _ = fcm_runs[10.0]
check(f"[B-fcm] fuzzifier 10: every degree of belonging lies between 0.25 "
      f"and 0.75 (smallest {U_10.min():.2f}, largest {U_10.max():.2f})",
      np.all((U_10 > 0.25) & (U_10 < 0.75)))


# ---- [B-plot] CSVs and preview
print("[B-plot] thinned scatter, boundary curves, the line through the "
      "centroids")
idx = np.arange(0, m, 18)
with open(OUT_DIR / "softclustering_points_plot.csv", "w") as fh:
    fh.write("tmin,tmax,cluster,confident,dobcold\n")
    for i in idx:
        fh.write(f"{X[i, 0]:.1f},{X[i, 1]:.1f},{int(c_g[i])},"
                 f"{int(U_g[i].max() >= 0.8)},{U_g[i, 0]:.4f}\n")
check(f"[B-plot] every 18th day: {len(idx)} of the {m} days are drawn, "
      f"{int(np.sum(U_g[idx].max(axis=1) < 0.8))} of them ambiguous",
      len(idx) == 21)
with open(OUT_DIR / "softclustering_centroids.csv", "w") as fh:
    fh.write("cluster,tmin,tmax\n")
    for j in range(K):
        fh.write(f"{j},{W_km[j, 0]:.4f},{W_km[j, 1]:.4f}\n")

# the k-means decision boundary: the bisector of the two centroids
mid = W_km.mean(axis=0)
axis_vec = W_km[1] - W_km[0]
perp = np.array([-axis_vec[1], axis_vec[0]]) / np.linalg.norm(axis_vec)
with open(OUT_DIR / "softclustering_bisector.csv", "w") as fh:
    fh.write("tmin,tmax\n")
    for t in (-30.0, 30.0):
        q = mid + t * perp
        fh.write(f"{q[0]:.4f},{q[1]:.4f}\n")

# the curve of equal GMM degrees of belonging: the zero level set of
# the log posterior ratio, one root per column of a grid over the window
XMIN, XMAX, YMIN, YMAX = -8.0, 23.0, 0.0, 40.0
tmin_grid = np.linspace(XMIN, XMAX, 321)
tmax_grid = np.linspace(YMIN, YMAX, 841)
contour = []
roots_per_column = []
for t in tmin_grid:
    col = np.column_stack([np.full_like(tmax_grid, t), tmax_grid])
    U_col, _ = gmm_posterior(col, p_g, mu_g, C_g)
    ratio = np.log(U_col[:, 0]) - np.log(U_col[:, 1])
    sign_change = np.nonzero(np.sign(ratio[:-1]) != np.sign(ratio[1:]))[0]
    roots_per_column.append(len(sign_change))
    for i in sign_change:
        frac = ratio[i] / (ratio[i] - ratio[i + 1])
        contour.append((t, tmax_grid[i] + frac * (tmax_grid[i + 1] - tmax_grid[i])))
contour = np.array(contour)
check("[B-plot] within the window of panel (a) the curve of equal degrees of "
      "belonging has at most one point above each morning minimum",
      max(roots_per_column) <= 1)
with open(OUT_DIR / "softclustering_contour.csv", "w") as fh:
    fh.write("tmin,tmax\n")
    for q in contour:
        fh.write(f"{q[0]:.4f},{q[1]:.4f}\n")
U_on, _ = gmm_posterior(contour, p_g, mu_g, C_g)
check(f"[B-plot] the curve of equal degrees of belonging has {len(contour)} "
      f"vertices, each with both degrees of belonging within 0.02 of 0.5",
      np.all(np.abs(U_on[:, 0] - 0.5) < 0.02))

# the line through the two k-means centroids: s = 0 cold, s = 1 warm
s = np.linspace(-0.5, 1.5, 401)
line = W_km[0] + s[:, None] * axis_vec
u_gmm = gmm_posterior(line, p_g, mu_g, C_g)[0][:, 0]
u_sig = constrained_dob(line, W_km, 4.0)[:, 0]
u_fcm = fcm_dob(line, W_f, 2.0)[:, 0]
u_km = (s < 0.5).astype(float)
with open(OUT_DIR / "softclustering_segment.csv", "w") as fh:
    fh.write("s,kmeans,gmm,sigma4,fcm\n")
    for row in zip(s, u_km, u_gmm, u_sig, u_fcm):
        fh.write(",".join(f"{v:.4f}" for v in row) + "\n")
check("[B-plot] along the line, the GMM posterior and the constrained GMM "
      "give the cold cluster a degree of belonging of 1 at s = -0.5 and 0 "
      "at s = 1.5 to within 0.02",
      all(abs(u[0] - 1) < 0.02 and abs(u[-1]) < 0.02
          for u in (u_gmm, u_sig)))
check(f"[B-plot] fuzzy c-means with fuzzifier 2 does not: its degree of "
      f"belonging is {u_fcm[0]:.2f} at s = -0.5 and {u_fcm[-1]:.2f} at "
      f"s = 1.5, since it decays like one over the squared distance",
      0.8 < u_fcm[0] < 0.95 and 0.05 < u_fcm[-1] < 0.2)
check("[B-plot] the constrained GMM with sigma^2 = 4 crosses 0.5 at the "
      "midpoint s = 0.5, where k-means jumps",
      abs(s[np.argmin(np.abs(u_sig - 0.5))] - 0.5) < 0.01)

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(9.6, 4.0))
for j, mk, lab in ((0, "o", "cold days"), (1, "s", "warm days")):
    for conf, face in ((1, "black"), (0, "none")):
        sel = idx[(c_g[idx] == j)
                  & ((U_g[idx].max(axis=1) >= 0.8) == bool(conf))]
        if len(sel) == 0:
            continue
        ax1.scatter(X[sel, 0], X[sel, 1], marker=mk, facecolors=face,
                    edgecolors="black", s=30,
                    label=f"{lab}, {'largest DOB >= 0.8' if conf else 'largest DOB < 0.8'}")
ax1.plot(contour[:, 0], contour[:, 1], "k-", lw=1.2,
         label="equal DOBs under the GMM")
bis = np.array([mid + t * perp for t in (-30, 30)])
ax1.plot(bis[:, 0], bis[:, 1], "k--", lw=1.0, label="k-means decision boundary")
ax1.plot(line[:, 0], line[:, 1], "k:", lw=0.8, label="line of panel (b)")
ax1.scatter(W_km[:, 0], W_km[:, 1], marker="x", color="black", s=60,
            label="k-means centroids")
ax1.set_xlim(XMIN, XMAX)
ax1.set_ylim(YMIN, YMAX)
ax1.set_xlabel("morning minimum temperature (°C)")
ax1.set_ylabel("maximum daytime temperature (°C)")
ax1.set_title("(a) every 18th day of 2024 and the two boundaries")
ax1.legend(frameon=False, fontsize=7, loc="upper left")
ax2.plot(s, u_km, "k--", lw=1.0, label="k-means (hard)")
ax2.plot(s, u_gmm, "k-", lw=1.4, label="GMM posterior")
ax2.plot(s, u_sig, "k-.", lw=1.0, label="GMM with sigma^2 = 4")
ax2.plot(s, u_fcm, "k:", lw=1.4, label="fuzzy c-means, fuzzifier 2")
ax2.set_xlabel("position s on the line through the centroids")
ax2.set_ylabel("degree of belonging to the cold cluster")
ax2.set_title("(b) soft and hard degrees of belonging")
ax2.legend(frameon=False, fontsize=8)
fig.tight_layout()
fig.savefig(OUT_DIR / "softclustering.png", dpi=110)

n_ok = sum(ok for _, ok in report)
print(f"\n{n_ok}/{len(report)} checks pass")
if n_ok != len(report):
    raise SystemExit(1)
