{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3",
   "language": "python"
  },
  "language_info": {
   "name": "python"
  },
  "colab": {
   "name": "kernelridgeregression.ipynb"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# kernel ridge regression \u2014 Python demo\n\nNumerical companion to the entry [kernel ridge regression](https://dictionaryofml.org/terms/kernelridgeregression.html) of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org/): it recomputes what the entry states and prints one line per check.\n\nKernel ridge regression (KRR) is RERM over the RKHS H_k with the squared error loss,\n\nRequires NumPy and Matplotlib only, and uses fixed seeds, so the printed numbers reproduce exactly. Generated from [`pythondemos/kernelridgeregression.py`](https://dictionaryofml.org/terms/kernelridgeregression.py); CC BY 4.0."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "# Notebook shim: the script resolves output paths relative to __file__,\n# which a notebook kernel does not define; everything lands in the\n# working directory instead.\nimport os\n__file__ = os.path.join(os.getcwd(), \"kernelridgeregression.py\")\nos.makedirs(\"pythondemos\", exist_ok=True)"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "\"\"\"\nkernelridgeregression.py \u2014 numerical companion to the glossary entry\n'kernel ridge regression'.\n\nPurpose\n-------\nKernel ridge regression (KRR) is RERM over the RKHS H_k with the squared\nerror loss,\n\n    min_{h in H_k}  (1/m) sum_r (y^(r) - h(x^(r)))^2  +  alpha ||h||_{H_k}^2 ,\n\nwhose representer-theorem solution h_hat = sum_r beta_r k(x^(r), .) has the\nclosed-form expansion coefficients  beta_hat = (K + alpha m I)^{-1} y  with\nGram matrix K_rs = k(x^(r), x^(s)).  The demo verifies the closed form on a\none-dimensional trainset, the reduction to ridge regression for the linear\nkernel, and the data augmentation interpretation (Bishop 1995) in the\nsetting of the entry: the kernel k(x, x') = x^T C^{-1} x' on two features,\nwhose RKHS is R^2 with the modified inner product, so that KRR is ridge\nregression with the penalty alpha w^T C w and the augmentation perturbs the\nraw feature vectors with covariance alpha C, i.e., within the ellipse of\nradius sqrt(alpha) in the norm induced by the kernel.  Self-contained\n(numpy/matplotlib only), fixed seeds.\n\nBlocks\n------\n[B-data]     m = 20 scalar features x^(r) uniform in [-3, 3] (seed 0),\n             labels y^(r) = sin(2 x^(r)) + 0.15 * noise.\n[B-closed]   Gaussian kernel (sigma = 0.7), alpha = 1e-2: closed-form\n             beta_hat solves (K + alpha m I) beta = y up to rounding\n             error; training MSE below the noise level; linear ridge\n             regression on [x, 1] with the same alpha fits a straight\n             line with a training MSE more than ten times larger.\n[B-linker]   Linear kernel k(x, x') = x^T x' on a two-feature trainset:\n             KRR predictions coincide with ridge regression on the raw\n             feature vectors (same alpha).\n[B-metric]   Kernel k(x, x') = x^T C^{-1} x' with C = R diag(1.5^2, 0.75^2)\n             R^T, R the rotation by 30 degrees (eigenvalues 1.5^2, 0.75^2,\n             eigenvectors at 30 degrees to the feature axes):\n             the KRR prediction sum_r beta_r k(x^(r), x) equals w_hat^T x\n             with w_hat = (X^T X + alpha m C)^{-1} X^T y = C^{-1} X^T\n             beta_hat (ridge regression with penalty alpha w^T C w); the\n             average squared error loss over perturbed copies x^(r) +\n             eps, eps ~ N(0, alpha C), exceeds the original loss by\n             exactly alpha w^T C w (Monte Carlo); linear regression on a\n             large augmented trainset recovers w_hat; the random function\n             k(eps, .) has covariance alpha k(x, x') (Monte Carlo, four\n             standard errors); on the ellipse (x - x^(r))^T C^{-1}\n             (x - x^(r)) = alpha the kernel norm of the perturbation is\n             sqrt(alpha).\n[B-figure]   Two-feature trainset of six data points for the entry's\n             augmentation figure (alpha = 1/2): the one-standard-deviation\n             ellipse of the perturbation around each data point, i.e.,\n             the ball of radius sqrt(alpha) in the kernel norm, six\n             perturbed copies per point, and the contour lines w_hat^T x\n             = c of the hypothesis learned from the six labelled points\n             (closed form with the figure's alpha and C), clipped to the\n             axis box, with their level values; on one ellipse the two\n             principal axes sqrt(alpha lambda_j) u^(j), the eigenvectors\n             u^(j) of C scaled by the square roots of alpha times its\n             eigenvalues lambda_j, which end on the ellipse.\n\nOutputs\n-------\nkernelridgeregression_points.csv       : the 1-D trainset, columns x,y.\nkernelridgeregression_curves.csv       : dense grid, columns x,krr,lin\n                                         (KRR fit and linear-ridge fit).\nkernelridgeregression_aug_points.csv   : the six original feature\n                                         vectors, columns x1,x2.\nkernelridgeregression_aug_copies.csv   : the perturbed copies, x1,x2.\nkernelridgeregression_aug_ellipses.csv : the six ellipses as polylines\n                                         separated by nan rows, x1,x2.\nkernelridgeregression_aug_contours.csv : contour lines of w_hat^T x as\n                                         segments separated by nan rows.\nkernelridgeregression_aug_labels.csv   : x1,x2,label \u2014 the right-hand end\n                                         of each contour line and its level.\nkernelridgeregression_aug_axis1.csv,\nkernelridgeregression_aug_axis2.csv    : the two principal axes of one\n                                         ellipse, center and tip, x1,x2.\nkernelridgeregression.png              : preview (checking only).\n\"\"\"\n\nimport numpy as np\nimport matplotlib\n\nmatplotlib.use(\"Agg\")\nimport matplotlib.pyplot as plt\n\nfrom pathlib import Path\n\nOUT_DIR = Path(__file__).parent\n\nreport = []\n\n\ndef check(name, ok):\n    report.append((name, bool(ok)))\n    print(f\"  [{'ok' if ok else 'FAIL'}] {name}\")\n\n\ndef krr_coefficients(K, y, alpha):\n    \"\"\"beta_hat = (K + alpha m I)^{-1} y.\"\"\"\n    m = len(y)\n    return np.linalg.solve(K + alpha * m * np.eye(m), y)\n\n\ndef ridge(Xm, ym, alpha):\n    \"\"\"w_hat = (X^T X + alpha m I)^{-1} X^T y (the ridgeregression entry).\"\"\"\n    mm, dd = Xm.shape\n    return np.linalg.solve(Xm.T @ Xm + alpha * mm * np.eye(dd), Xm.T @ ym)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-data]** m = 20 scalar features x^(r) uniform in [-3, 3] (seed 0), labels y^(r) = sin(2 x^(r)) + 0.15 * noise."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "rng = np.random.default_rng(0)\nm = 20\nx = np.sort(rng.uniform(-3.0, 3.0, m))\ny = np.sin(2.0 * x) + 0.15 * rng.standard_normal(m)\ncheck(\"[B-data] m = 20 noisy samples of sin(2x)\", m == 20)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-closed]** Gaussian kernel (sigma = 0.7), alpha = 1e-2: closed-form beta_hat solves (K + alpha m I) beta = y up to rounding error; training MSE below the noise level; linear ridge regression on [x, 1] with the same alpha fits a straight line with a training MSE more than ten times larger."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "SIGMA = 0.7\nALPHA = 1e-2\n\n\ndef gauss_kernel_1d(p, q):\n    return np.exp(-(p[:, None] - q[None, :]) ** 2 / (2.0 * SIGMA ** 2))\n\n\nK = gauss_kernel_1d(x, x)\nbeta = krr_coefficients(K, y, ALPHA)\n\n\ndef h_hat(p):\n    \"\"\"Representer expansion: h_hat(x) = sum_r beta_r k(x^(r), x).\"\"\"\n    return gauss_kernel_1d(p, x) @ beta\n\n\nresidual = float(np.linalg.norm((K + ALPHA * m * np.eye(m)) @ beta - y))\nmse_krr = float(np.mean((h_hat(x) - y) ** 2))\nX1 = np.c_[x, np.ones(m)]\nw_lin = ridge(X1, y, ALPHA)\nmse_lin = float(np.mean((X1 @ w_lin - y) ** 2))\ncheck(\"[B-closed] beta_hat solves (K + alpha m I) beta = y\", residual < 1e-10)\ncheck(f\"[B-closed] KRR training MSE {mse_krr:.4f} < 0.05\", mse_krr < 0.05)\ncheck(f\"[B-closed] linear ridge MSE {mse_lin:.3f} > 10 x KRR MSE\",\n      mse_lin > 10.0 * mse_krr)\n\ngrid = np.linspace(-3.2, 3.2, 201)\nwith open(OUT_DIR / \"kernelridgeregression_points.csv\", \"w\") as f:\n    f.write(\"x,y\\n\")\n    for xi, yi in zip(x, y):\n        f.write(f\"{xi:.4f},{yi:.4f}\\n\")\nwith open(OUT_DIR / \"kernelridgeregression_curves.csv\", \"w\") as f:\n    f.write(\"x,krr,lin\\n\")\n    for g, a, b in zip(grid, h_hat(grid), np.c_[grid, np.ones_like(grid)] @ w_lin):\n        f.write(f\"{g:.4f},{a:.4f},{b:.4f}\\n\")"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-linker]** Linear kernel k(x, x') = x^T x' on a two-feature trainset: KRR predictions coincide with ridge regression on the raw feature vectors (same alpha)."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "rng2 = np.random.default_rng(1)\nm2 = 30\nX2 = rng2.normal(size=(m2, 2))\ny2 = X2 @ np.array([1.5, -0.7]) + 0.2 * rng2.normal(size=m2)\nX2_new = rng2.normal(size=(50, 2))\nbeta_lin = krr_coefficients(X2 @ X2.T, y2, ALPHA)\npred_krr = X2_new @ X2.T @ beta_lin          # sum_r beta_r x^(r)^T x\npred_ridge = X2_new @ ridge(X2, y2, ALPHA)\ncheck(\"[B-linker] linear-kernel KRR predictions equal ridge regression on \"\n      \"the raw feature vectors\", np.allclose(pred_krr, pred_ridge))"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-metric]** Kernel k(x, x') = x^T C^{-1} x' with C = R diag(1.5^2, 0.75^2) R^T, R the rotation by 30 degrees (eigenvalues 1.5^2, 0.75^2, eigenvectors at 30 degrees to the feature axes): the KRR prediction sum_r beta_r k(x^(r), x) equals w_hat^T x with w_hat = (X^T X + alpha m C)^{-1} X^T y = C^{-1} X^T beta_hat (ridge regression with penalty alpha w^T C w); the average squared error loss over perturbed copies x^(r) + eps, eps ~ N(0, alpha C), exceeds the original loss by exactly alpha w^T C w (Monte Carlo); linear regression on a large augmented trainset recovers w_hat; the random function k(eps, .) has covariance alpha k(x, x') (Monte Carlo, four standard errors); on the ellipse (x - x^(r))^T C^{-1} (x - x^(r)) = alpha the kernel norm of the perturbation is sqrt(alpha)."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "THETA_C = np.pi / 6                                 # principal axes at 30 degrees\nR_C = np.array([[np.cos(THETA_C), -np.sin(THETA_C)],\n                [np.sin(THETA_C), np.cos(THETA_C)]])\nEIGVALS_C = np.array([1.5 ** 2, 0.75 ** 2])\nC = R_C @ np.diag(EIGVALS_C) @ R_C.T               # pd, eigenvectors = columns of R_C\nC_inv = np.linalg.inv(C)\n\n\ndef metric_kernel(P, Q):\n    return P @ C_inv @ Q.T\n\n\nbeta_m = krr_coefficients(metric_kernel(X2, X2), y2, ALPHA)\nw_metric = np.linalg.solve(X2.T @ X2 + ALPHA * m2 * C, X2.T @ y2)\ncheck(\"[B-metric] w_hat = (X^T X + alpha m C)^{-1} X^T y equals C^{-1} X^T beta_hat\",\n      np.allclose(w_metric, C_inv @ X2.T @ beta_m))\ncheck(\"[B-metric] KRR predictions equal w_hat^T x (ridge regression with \"\n      \"penalty alpha w^T C w)\",\n      np.allclose(metric_kernel(X2_new, X2) @ beta_m, X2_new @ w_metric))\n\nw_probe = rng2.normal(size=2)\nr = 0\nn_mc = 200_000\neps = rng2.multivariate_normal(np.zeros(2), ALPHA * C, size=n_mc)\nloss_orig = (y2[r] - X2[r] @ w_probe) ** 2\nloss_pert = np.mean((y2[r] - (X2[r] + eps) @ w_probe) ** 2)\nexcess = float(ALPHA * w_probe @ C @ w_probe)     # variance of w^T eps\nstd_err = np.sqrt(2.0 * excess ** 2 + 4.0 * loss_orig * excess) / np.sqrt(n_mc)\nprint(f\"  excess loss {loss_pert - loss_orig:.5f}, alpha w^T C w {excess:.5f}, \"\n      f\"standard error {std_err:.5f}\")\ncheck(\"[B-metric] average loss over perturbed copies = original loss + \"\n      \"alpha w^T C w within four standard errors (Monte Carlo)\",\n      abs(loss_pert - loss_orig - excess) < 4.0 * std_err)\nn_fit = 3000\nX_aug = np.repeat(X2, n_fit, axis=0) \\\n    + rng2.multivariate_normal(np.zeros(2), ALPHA * C, size=m2 * n_fit)\ny_aug = np.repeat(y2, n_fit)                        # labels left unchanged\nw_aug = np.linalg.lstsq(X_aug, y_aug, rcond=None)[0]\ncheck(\"[B-metric] linear regression on the augmented trainset recovers w_hat\",\n      np.allclose(w_aug, w_metric, atol=5e-2))\n# covariance of the random function k(eps, .)\npairs = rng2.normal(size=(4, 2, 2))\ncov_ok = True\nfor xa, xb in pairs:\n    ka, kb = eps @ C_inv @ xa, eps @ C_inv @ xb        # k(eps, xa), k(eps, xb)\n    emp = np.mean(ka * kb)\n    k_ab = float(xa @ C_inv @ xb)\n    k_aa, k_bb = float(xa @ C_inv @ xa), float(xb @ C_inv @ xb)\n    std_err = ALPHA * np.sqrt(k_ab ** 2 + k_aa * k_bb) / np.sqrt(n_mc)\n    cov_ok &= abs(emp - ALPHA * k_ab) < 4.0 * std_err\ncheck(\"[B-metric] E{k(eps, x) k(eps, x')} = alpha k(x, x') within four \"\n      \"standard errors (Monte Carlo, 4 pairs)\", bool(cov_ok))\ntheta = np.linspace(0, 2 * np.pi, 61)\nL = np.linalg.cholesky(C)\n\n\ndef ellipse(alpha):\n    \"\"\"The contour d^T C^{-1} d = alpha, i.e., k(d, d) = alpha.\"\"\"\n    return np.sqrt(alpha) * (L @ np.stack([np.cos(theta), np.sin(theta)])).T\n\n\nring = ellipse(ALPHA)\ncheck(\"[B-metric] on the one-standard-deviation ellipse the kernel norm of \"\n      \"the perturbation is sqrt(alpha)\",\n      np.allclose(np.sqrt(np.einsum(\"ij,jk,ik->i\", ring, C_inv, ring)),\n                  np.sqrt(ALPHA)))"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**[B-figure]** Two-feature trainset of six data points for the entry's augmentation figure (alpha = 1/2): the one-standard-deviation ellipse of the perturbation around each data point, i.e., the ball of radius sqrt(alpha) in the kernel norm, six perturbed copies per point, and the contour lines w_hat^T x = c of the hypothesis learned from the six labelled points (closed form with the figure's alpha and C), clipped to the axis box, with their level values; on one ellipse the two principal axes sqrt(alpha lambda_j) u^(j), the eigenvectors u^(j) of C scaled by the square roots of alpha times its eigenvalues lambda_j, which end on the ellipse."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "ALPHA_FIG = 0.5\nrng3 = np.random.default_rng(3)\nP = np.array([[-1.6, 0.8], [-0.4, -0.9], [0.3, 1.1], [1.2, 0.2],\n              [2.0, -0.7], [-2.3, -0.3]])\ncopies = np.vstack([p + rng3.multivariate_normal(np.zeros(2), ALPHA_FIG * C, size=6)\n                    for p in P])\nring_fig = ellipse(ALPHA_FIG)\nwith open(OUT_DIR / \"kernelridgeregression_aug_points.csv\", \"w\") as f:\n    f.write(\"x1,x2\\n\")\n    for p in P:\n        f.write(f\"{p[0]:.3f},{p[1]:.3f}\\n\")\nwith open(OUT_DIR / \"kernelridgeregression_aug_copies.csv\", \"w\") as f:\n    f.write(\"x1,x2\\n\")\n    for c in copies:\n        f.write(f\"{c[0]:.3f},{c[1]:.3f}\\n\")\nwith open(OUT_DIR / \"kernelridgeregression_aug_ellipses.csv\", \"w\") as f:\n    f.write(\"x1,x2\\n\")\n    for p in P:\n        for e in ring_fig + p:\n            f.write(f\"{e[0]:.3f},{e[1]:.3f}\\n\")\n        f.write(\"nan,nan\\n\")\n# labels for the six data points, the learned hypothesis, and its contour lines\ny_fig = P @ np.array([0.8, 1.2]) + 0.3 * rng3.normal(size=len(P))\nw_fig = np.linalg.solve(P.T @ P + ALPHA_FIG * len(P) * C, P.T @ y_fig)\nbeta_fig = krr_coefficients(metric_kernel(P, P), y_fig, ALPHA_FIG)\ncheck(\"[B-figure] w_hat on the six data points equals C^{-1} X^T beta_hat\",\n      np.allclose(w_fig, C_inv @ P.T @ beta_fig))\nBOX = (-3.5, 3.5, -1.9, 2.1)                       # the figure's axis limits\n\n\ndef clip_line(w, c, box):\n    \"\"\"Endpoints of the line w^T x = c inside the box (None if it misses).\"\"\"\n    x1min, x1max, x2min, x2max = box\n    pts = []\n    for x1 in (x1min, x1max):                       # crossings of the sides\n        if abs(w[1]) > 1e-12:\n            x2 = (c - w[0] * x1) / w[1]\n            if x2min - 1e-9 <= x2 <= x2max + 1e-9:\n                pts.append((x1, x2))\n    for x2 in (x2min, x2max):                       # crossings of top/bottom\n        if abs(w[0]) > 1e-12:\n            x1 = (c - w[1] * x2) / w[0]\n            if x1min - 1e-9 <= x1 <= x1max + 1e-9:\n                pts.append((x1, x2))\n    pts = sorted(set((round(a, 6), round(b, 6)) for a, b in pts))\n    return (pts[0], pts[-1]) if len(pts) >= 2 else None\n\n\ncorners = np.array([[BOX[0], BOX[2]], [BOX[0], BOX[3]], [BOX[1], BOX[2]], [BOX[1], BOX[3]]])\nvals = corners @ w_fig\nstep = 1.0 if vals.max() - vals.min() < 6.0 else 2.0\nlevels = np.arange(np.ceil(vals.min()), np.floor(vals.max()) + 1e-9, step)\nsegments = [(c, clip_line(w_fig, c, BOX)) for c in levels]\nsegments = [(c, seg) for c, seg in segments if seg is not None]\nwith open(OUT_DIR / \"kernelridgeregression_aug_contours.csv\", \"w\") as f:\n    f.write(\"x1,x2\\n\")\n    for c, (p0, p1) in segments:\n        f.write(f\"{p0[0]:.3f},{p0[1]:.3f}\\n{p1[0]:.3f},{p1[1]:.3f}\\nnan,nan\\n\")\nwith open(OUT_DIR / \"kernelridgeregression_aug_labels.csv\", \"w\") as f:\n    f.write(\"x1,x2,label\\n\")\n    for c, (p0, p1) in segments:\n        p = p1 if p1[0] >= p0[0] else p0            # the right-hand endpoint\n        f.write(f\"{p[0]:.3f},{p[1]:.3f},{c:g}\\n\")\nprint(f\"  hypothesis on the figure's trainset: w_hat = ({w_fig[0]:.2f}, \"\n      f\"{w_fig[1]:.2f}); contour levels {levels}\")\n# principal axes of the ellipse: eigenvectors of C, semi-axes sqrt(alpha lambda_j)\nlam, U = np.linalg.eigh(C)\norder = np.argsort(lam)[::-1]\nlam, U = lam[order], U[:, order]\nU = U * np.sign(U[0, :])                            # orient each axis rightwards\nsemi = np.sqrt(ALPHA_FIG * lam)\np_axes = P[0]                                       # the ellipse that carries the arrows\nfor jx in range(2):\n    tip = p_axes + semi[jx] * U[:, jx]\n    with open(OUT_DIR / f\"kernelridgeregression_aug_axis{jx + 1}.csv\", \"w\") as f:\n        f.write(\"x1,x2\\n\")\n        f.write(f\"{p_axes[0]:.3f},{p_axes[1]:.3f}\\n{tip[0]:.3f},{tip[1]:.3f}\\n\")\nquad = [float((semi[jx] * U[:, jx]) @ C_inv @ (semi[jx] * U[:, jx])) for jx in range(2)]\ncheck(\"[B-figure] the principal axes sqrt(alpha lambda_j) u^(j) of C end on \"\n      \"the ellipse d^T C^{-1} d = alpha\", np.allclose(quad, ALPHA_FIG))\ncheck(\"[B-figure] the eigenvectors of C are the axes at 30 degrees\",\n      np.allclose(np.abs(U[:, 0] @ R_C[:, 0]), 1.0) and np.isclose(lam[0], EIGVALS_C[0]))\nprint(f\"  principal axes on x^(1) = ({p_axes[0]:.2f}, {p_axes[1]:.2f}): tips \"\n      f\"({(p_axes + semi[0] * U[:, 0])[0]:.2f}, {(p_axes + semi[0] * U[:, 0])[1]:.2f}) and \"\n      f\"({(p_axes + semi[1] * U[:, 1])[0]:.2f}, {(p_axes + semi[1] * U[:, 1])[1]:.2f}); \"\n      f\"semi-axes {semi[0]:.2f}, {semi[1]:.2f}\")\ncheck(\"[B-figure] six data points, 36 perturbed copies, six ellipses, \"\n      f\"{len(segments)} contour lines written\",\n      len(P) == 6 and len(copies) == 36 and len(segments) >= 4)\n\n# ---- preview (checking only)\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 4))\nax1.plot(grid, h_hat(grid), \"k-\", lw=1.8, label=\"KRR $\\\\hat{h}$\")\nax1.plot(grid, np.c_[grid, np.ones_like(grid)] @ w_lin, \"k--\", lw=1.2,\n         label=\"linear ridge regression\")\nax1.plot(x, y, \"ko\", ms=4, mfc=\"none\", label=\"trainset\")\nax1.set_xlabel(\"feature $x$\")\nax1.set_ylabel(\"label $y$\")\nax1.set_title(\"Gaussian-kernel KRR fits sin(2x); linear ridge cannot\")\nax1.legend(frameon=False, fontsize=8)\nfor p in P:\n    e = ring_fig + p\n    ax2.plot(e[:, 0], e[:, 1], color=\"0.5\", lw=1)\nfor jx in range(2):\n    tip = p_axes + semi[jx] * U[:, jx]\n    ax2.annotate(\"\", xy=tip, xytext=p_axes, arrowprops=dict(arrowstyle=\"->\", lw=1.2))\nfor c, (p0, p1) in segments:\n    ax2.plot([p0[0], p1[0]], [p0[1], p1[1]], color=\"0.6\", lw=0.8, ls=\"--\")\n    ax2.annotate(f\"{c:g}\", (p1 if p1[0] >= p0[0] else p0), fontsize=7,\n                 color=\"0.4\", ha=\"left\", va=\"bottom\")\nax2.scatter(copies[:, 0], copies[:, 1], marker=\"s\", facecolors=\"none\",\n            edgecolors=\"tab:red\", s=16, label=\"perturbed copy\")\nax2.scatter(P[:, 0], P[:, 1], marker=\"o\", color=\"tab:blue\", s=30,\n            label=\"original data point\")\nax2.set_aspect(\"equal\")\nax2.set_xlabel(\"feature $x_1$\")\nax2.set_ylabel(\"feature $x_2$\")\nax2.set_title(\"Kernel-norm balls of radius $\\\\sqrt{\\\\alpha}$ and contours of $\\\\hat{w}^T x$\")\nax2.legend(frameon=False, fontsize=8)\nfig.tight_layout()\nfig.savefig(OUT_DIR / \"kernelridgeregression.png\", dpi=110)\n\nn_ok = sum(ok for _, ok in report)\nprint(f\"\\n{n_ok}/{len(report)} checks pass\")\nif n_ok != len(report):\n    raise SystemExit(1)"
  }
 ]
}