kernelridgeregression.py, which recomputes what the entry states: the script block by block · open in ColabDictionary of Applied Machine Learning: 114 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here
Updated on 2026-11-04
Readiness 96/100
See also kernel method data augmentation
Kernel ridge regression is a kernel method for regression that applies ridge regression to transformed feature vectors constructed from a kernel, and thereby learns a nonlinear hypothesis in closed form. It performs regularized empirical risk minimization (RERM) over the reproducing kernel Hilbert space (RKHS) of the kernel, combining the squared error loss with a squared-norm penalty term. By the representer theorem, the learned hypothesis is a weighted sum of kernel evaluations at the feature vectors in the training set, and the coefficients are obtained in closed form by solving a single system of linear equations. Like ridge regression, the method equals empirical risk minimization (ERM) on an augmented training set: the transformed feature vectors are perturbed by realizations of a zero-mean Gaussian process (GP) whose covariance function is the kernel scaled by the regularization parameter. For the kernel that is a modified inner product of the feature space, kernel ridge regression is ridge regression with a matching penalty term. The perturbed copies of a data point then fill an ellipse whose shape the kernel sets, the circle of ridge regression for the standard inner product.
kernelridgeregression.py, which recomputes what the entry states: the script block by block · open in ColabThe temperature over the course of a day rises and falls, so predicting it from the time of the day is a regression problem whose relation between feature and label is nonlinear. Kernel ridge regression solves such regression problems by applying ridge regression to transformed feature vectors constructed from a kernel, and thereby learns a nonlinear hypothesis in closed form (Hastie et al., 2009, Sect. 5.8.2; Schölkopf and Smola, 2002). Fig. 1 shows a training set with a periodic relation between feature and label. With the Gaussian kernel $k\big(x,x'\big) = \exp\big( - (x - x')^{2} / (2 \sigma^{2}) \big)$ of bandwidth $\sigma = 0.7$, the learned hypothesis follows the data points smoothly, with a training mean squared error (MSE) of $0.013$, while ridge regression on the raw feature can only learn a linear hypothesis, with a training MSE of $0.36$.
B-dataFormally, kernel ridge regression learns a hypothesis from the reproducing kernel Hilbert space (RKHS) $\mathcal{H}_{k}$ of a kernel $k: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ by minimizing the penalized average squared error loss on a training set $\mathcal{D}^{(\rm train)} = \big\{ \left( {\mathbf x}^{(r)},y^{(r)} \right) \big\}_{r=1}^{m}$ with numeric labels $y^{(r)} \in \mathbb{R}$, \begin{equation} \label{eq:krr_rerm} \widehat{h} = \mathop{\mathrm{arg\,min}}_{h \in \mathcal{H}_{k}} \frac{1}{m} \sum_{r=1}^{m} \big( y^{(r)} - h\big({\mathbf x}^{(r)}\big) \big)^{2} + \alpha \left\lVert h \right\rVert_{\mathcal{H}_{k}}^{2} \text{.} \end{equation} The penalty term is the scaled squared RKHS norm $\alpha \left\lVert h \right\rVert_{\mathcal{H}_{k}}^{2}$ with a regularization parameter $\alpha > 0$. The resulting method is an instance of regularized empirical risk minimization (RERM), as is ridge regression, whose penalty term is the scaled squared Euclidean norm $\alpha \left\lVert {\mathbf w} \right\rVert_{2}^{2}$ of the model parameters (Hastie et al., 2009, Sect. 5.8.2). The average squared error loss is a convex function of $h$, and the squared norm is strictly convex. The kernel ridge regression problem \(\eqref{eq:krr_rerm}\) therefore has a unique minimizer $\widehat{h}$ for any training set, as does ridge regression.
By the representer theorem (see kernel method), the minimizer of the kernel ridge regression problem \(\eqref{eq:krr_rerm}\) has the form $\widehat{h} = \sum_{r=1}^{m} \beta_{r} k\big({\mathbf x}^{(r)},\cdot\big)$ with a coefficient vector $\boldsymbol{\beta} = \big(\beta_{1}, \ldots, \beta_{m}\big)^{\top}$. Inserting this form into the kernel ridge regression problem \(\eqref{eq:krr_rerm}\) yields the finite-dimensional optimization problem \[ \min_{\boldsymbol{\beta} \in \mathbb{R}^{m}} \frac{1}{m} \left\lVert {\mathbf y} - \mathbf{K} \boldsymbol{\beta} \right\rVert_{2}^{2} + \alpha\, \boldsymbol{\beta}^{\top} \mathbf{K} \boldsymbol{\beta} \text{,} \] with the label vector ${\mathbf y} = \big(y^{(1)}, \ldots, y^{(m)}\big)^{\top}$ and the Gram matrix $\mathbf{K}$ with entries $K_{r,r'} = k\big({\mathbf x}^{(r)},{\mathbf x}^{(r')}\big)$ (see kernel).
B-closedBy the zero-gradient condition, a
coefficient vector solving this optimization problem satisfies
the system of linear equations
\[
\mathbf{K}\big( \big(\mathbf{K} + \alpha m \mathbf{I}\big) \boldsymbol{\beta} - {\mathbf y} \big) = \mathbf{0} \text{.}
\]
Since $\mathbf{K}$ is positive semi-definite (psd) and $\alpha > 0$, the matrix
$\mathbf{K} + \alpha m \mathbf{I}$ is invertible, and a
minimizer is obtained in closed form as
\begin{equation}
\label{eq:krr_closedform}
\widehat{\boldsymbol{\beta}}
= \big( \mathbf{K} + \alpha m \mathbf{I} \big)^{-1} {\mathbf y}
\text{.}
\end{equation}
The prediction for a data point with feature vector
${\mathbf x}$ is
\[
\widehat{h}({\mathbf x}) = \sum_{r=1}^{m} \widehat{\beta}_{r} k\big({\mathbf x}^{(r)},{\mathbf x}\big) \text{.}
\]
Training is a one-shot computation: kernel ridge regression
requires no iterative optimization method (and hence no fixed-point
interpretation), only the unique solution of the
$m \times m$ system of linear equations
with the invertible matrix $\mathbf{K} + \alpha m \mathbf{I}$,
which is the closed form \(\eqref{eq:krr_closedform}\).
pythondemos/kernelridgeregression.py
Like ridge regression, kernel ridge regression has an interpretation as a form of data augmentation. Training with noise added to the feature vectors is equivalent to adding a penalty term (Bishop, 1995). Replacing each data point by a probability distribution centered at it is the vicinal form of empirical risk minimization (ERM) (Chapelle et al., 2001), and its effect on kernel methods is a regularization of the learned hypothesis (Dao et al., 2019).
The following example makes the equivalence explicit for a linear kernel on the plane, so that the perturbed copies can be drawn. Consider data points with two features, the feature space $\mathcal{X} = \mathbb{R}^{2}$, and the kernel \begin{equation} \label{eq:krr_metric_kernel} k\big({\mathbf x},{\mathbf x}'\big) := {\mathbf x}^{\top} {\mathbf C}^{-1} {\mathbf x}' \text{,} \end{equation} with a positive definite (pd) matrix ${\mathbf C} \in \mathbb{R}^{2 \times 2}$. This kernel is an inner product of $\mathbb{R}^{2}$ with a modified metric: the squared norm $k\big({\mathbf x},{\mathbf x}\big) = {\mathbf x}^{\top} {\mathbf C}^{-1} {\mathbf x}$ scales the two features differently and couples them, and its unit ball is an ellipse instead of the circle of the Euclidean norm. The RKHS $\mathcal{H}_{k}$ of this kernel is $\mathbb{R}^{2}$ with this inner product. The transformed feature vector $k\big({\mathbf x},\cdot\big)$ is the linear function ${\mathbf x}' \mapsto {\mathbf x}^{\top} {\mathbf C}^{-1} {\mathbf x}'$, every hypothesis in $\mathcal{H}_{k}$ is a linear model $h^{({\mathbf w})}({\mathbf x}) = {\mathbf w}^{\top} {\mathbf x}$ with model parameters ${\mathbf w} \in \mathbb{R}^{2}$, and its RKHS norm is $\left\lVert h^{({\mathbf w})} \right\rVert_{\mathcal{H}_{k}}^{2} = {\mathbf w}^{\top} {\mathbf C} {\mathbf w}$.
The kernel ridge regression problem \(\eqref{eq:krr_rerm}\) therefore reads \begin{equation} \label{eq:krr_metric_ridge} \widehat{{\mathbf w}} = \mathop{\mathrm{arg\,min}}_{{\mathbf w} \in \mathbb{R}^{2}} \frac{1}{m} \sum_{r=1}^{m} \big( y^{(r)} - {\mathbf w}^{\top} {\mathbf x}^{(r)} \big)^{2} + \alpha \, {\mathbf w}^{\top} {\mathbf C} {\mathbf w} \text{,} \end{equation} which is ridge regression with the penalty term $\alpha \left\lVert {\mathbf w} \right\rVert_{2}^{2}$ replaced by $\alpha \, {\mathbf w}^{\top} {\mathbf C} {\mathbf w}$. Its unique minimizer is $\widehat{{\mathbf w}} = \big( {\mathbf X}^{\top} {\mathbf X} + \alpha m {\mathbf C} \big)^{-1} {\mathbf X}^{\top} {\mathbf y}$ with the feature matrix ${\mathbf X} = \big({\mathbf x}^{(1)}, \ldots, {\mathbf x}^{(m)}\big)^{\top}$, and it coincides with ${\mathbf C}^{-1} {\mathbf X}^{\top} \widehat{\boldsymbol{\beta}}$ for the coefficients \(\eqref{eq:krr_closedform}\) with the Gram matrix $\mathbf{K} = {\mathbf X} {\mathbf C}^{-1} {\mathbf X}^{\top}$. For ${\mathbf C} = \mathbf{I}$, the kernel \(\eqref{eq:krr_metric_kernel}\) is the linear kernel and the penalized problem \(\eqref{eq:krr_metric_ridge}\) is ridge regression itself.
Adding the penalty term in the penalized problem \(\eqref{eq:krr_metric_ridge}\) is equivalent to ERM on an augmented training set, i.e., to computing the average squared error loss of $h^{({\mathbf w})}$ on it. This augmented training set is obtained from the original training set. For each data point $\left( {\mathbf x}^{(r)},y^{(r)} \right)$, $r = 1, \ldots, m$, the feature vector ${\mathbf x}^{(r)}$ is replaced by realizations of infinitely many independent and identically distributed (i.i.d.) random variables (RVs) ${\mathbf x}^{(r)} + {\boldsymbol \varepsilon}^{(r)}$ with Gaussian random variables (Gaussian RVs) ${\boldsymbol \varepsilon}^{(r)} \sim \mathcal{N}\left(\mathbf{0},\alpha {\mathbf C}\right)$. The label $y^{(r)}$ is left unchanged.
B-metricThe penalty term is exactly the amount by which the average squared error loss of $h^{({\mathbf w})}$ over the perturbed copies exceeds its squared error loss on the original data point. Since ${\mathbb{E}} \{ {\boldsymbol \varepsilon}^{(r)} \} = \mathbf{0}$, the increase is \[ {\mathbb{E}} \big\{ \big( {\mathbf w}^{\top} {\boldsymbol \varepsilon}^{(r)} \big)^{2} \big\} = {\mathbf w}^{\top} {\mathbb{E}} \big\{ {\boldsymbol \varepsilon}^{(r)} \big({\boldsymbol \varepsilon}^{(r)}\big)^{\top} \big\} {\mathbf w} = \alpha \, {\mathbf w}^{\top} {\mathbf C} {\mathbf w} \text{.} \] The equivalence is exact for the linear model $h^{({\mathbf w})}$, while for a nonlinear hypothesis it holds up to terms of higher order in the noise amplitude (Bishop, 1995). As in ridge regression, this increase of the average loss under data augmentation is an estimate for the generalization gap. Accordingly, linear regression on a large augmented training set recovers the predictions of kernel ridge regression.
The kernel determines the shape of these perturbations (see Fig. 2). The covariance matrix $\alpha {\mathbf C}$ of ${\boldsymbol \varepsilon}^{(r)}$ is the scaled inverse of the matrix in the kernel \(\eqref{eq:krr_metric_kernel}\). The one-standard-deviation contour of the perturbed copies is therefore \[ \big({\mathbf x} - {\mathbf x}^{(r)}\big)^{\top} {\mathbf C}^{-1} \big({\mathbf x} - {\mathbf x}^{(r)}\big) = \alpha \text{.} \] This contour is the set of feature vectors ${\mathbf x}$ with $k\big({\mathbf x} - {\mathbf x}^{(r)},{\mathbf x} - {\mathbf x}^{(r)}\big) = \alpha$, i.e., the ball of radius $\sqrt{\alpha}$ around ${\mathbf x}^{(r)}$ in the norm induced by the kernel. For ${\mathbf C} = \mathbf{I}$, this ball is a circle of radius $\sqrt{\alpha}$: the perturbations in the data augmentation interpretation of ridge regression have covariance matrix $\alpha \mathbf{I}$ and no preferred direction. For a general ${\mathbf C}$ it is an ellipse, so the ellipses drawn around the data points display the kernel. The size of a perturbation is measured by its kernel norm, and the penalty term $\alpha \, {\mathbf w}^{\top} {\mathbf C} {\mathbf w}$ is the average squared error loss that perturbations of kernel norm $\sqrt{\alpha}$ cause.
B-figureThe ellipses also display
the matrix ${\mathbf C}$ itself. Its eigenvalue decomposition (EVD)
${\mathbf C} = \sum_{j=1}^{2} \lambda_{j} \mathbf{u}^{(j)} \big(\mathbf{u}^{(j)}\big)^{\top}$,
with eigenvalues $\lambda_{1} \geq \lambda_{2} > 0$ and
orthonormal eigenvectors $\mathbf{u}^{(1)}, \mathbf{u}^{(2)}$,
turns the contour into
\[
\sum_{j=1}^{2} \frac{\big( \big(\mathbf{u}^{(j)}\big)^{\top} \big({\mathbf x} - {\mathbf x}^{(r)}\big) \big)^{2}}{\alpha \lambda_{j}} = 1 \text{.}
\]
The principal axes of every ellipse therefore point along the
eigenvectors of ${\mathbf C}$, and the semi-axes have
lengths $\sqrt{\alpha \lambda_{1}}$ and
$\sqrt{\alpha \lambda_{2}}$. The perturbations spread most
along the eigenvector with the largest eigenvalue, and
the penalty term grows with the square of a displacement of
${\mathbf w}$ along $\mathbf{u}^{(j)}$, in proportion to
$\lambda_{j}$.
pythondemos/kernelridgeregression.py
See also: kernel method, ridge regression, kernel, reproducing kernel Hilbert space, regularized empirical risk minimization, squared error loss, regression, Gaussian process, data augmentation, feature transformation, penalty term.
@misc{dictml_kernelridgeregression,
author = {Jung, Alexander},
editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
title = {kernel ridge regression},
howpublished = {Dictionary of Applied Machine Learning (course edition)},
year = {2026},
doi = {10.5281/zenodo.21569296},
note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-04},
url = {https://dictionaryofml.org/terms/kernelridgeregression.html}
}