Dictionary of Applied Machine Learning: 114 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

kernel method

Updated on 2026-10-29

Readiness 100/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Markdown Cite this entry

See also Hilbert space support vector machine kernel ridge regression

A kernel method applies a linear method, such as a linear model or a linear classifier, to transformed feature vectors. The transformation is constructed from a kernel: each feature vector is mapped to a function with domain equal to the original feature space, and inner products between transformed feature vectors reduce to evaluations of a kernel. Kernel methods apply to any feature space on which a kernel is defined, including sets of text documents and graphs.

Figure produced by the Python demo of the entry kernel method
Figure written by the entry's Python demo kernelmethod.py, which recomputes what the entry states: the script block by block · open in Colab

Definition

Deciding whether an image shows a specific object (Lampert, 2009), or which topic a text document belongs to (Cristianini and Shawe-Taylor, 2000), are tasks for kernel methods. A kernel method is a machine learning (ML) method that applies a linear method, such as a linear model for regression or a linear classifier for classification, to transformed feature vectors that are constructed from a kernel (Lampert, 2009; Schölkopf and Smola, 2002).

Linear methods learn a hypothesis that combines the features of a data point linearly, e.g., $h({\mathbf x}) = {\mathbf w}^{\top} {\mathbf x}$ for the feature space $\mathcal{X} = \mathbb{R}^{d}$. Such a hypothesis predicts well only if the relation between the feature vectors and the labels of data points is approximately linear. Moreover, a linear method requires numeric feature vectors and is not directly applicable to data points from an arbitrary feature space $\mathcal{X}$, such as text documents or graphs.

B-linearBoth limitations can be mitigated by a feature transformation $\phi: \mathcal{X} \rightarrow \mathcal{H}$ from the original feature space $\mathcal{X}$ to a transformed feature space $\mathcal{H}$, chosen as a Hilbert space. Fig. 1 shows a binary classification problem with the first limitation: no straight line separates the two classes in the original feature space, so no linear classifier has a decision boundary that classifies them correctly. A suitable feature transformation makes a dataset "more linear": the relation between the transformed feature vectors $\phi({\mathbf x})$ and the labels is closer to linear in $\mathcal{H}$ than the relation between the original feature vectors ${\mathbf x}$ and the labels is in $\mathcal{X}$.

Figure 1 of the entry kernelmethod
Figure 1: Five data points characterized by feature vectors $\featurevec^{(\sampleidx)}$ and labels $\truelabel^{(\sampleidx)} \in \{ \circ, \square \}$ for $\sampleidx=1, \,\ldots, \,5$. The curved boundary indicates the original feature space $\featurespace$ (left); the parallelogram indicates a section of the transformed feature space $\hilbertspace$ (right), which is a linear space (see Hilbert space). With these feature vectors, there is no way to separate the two classes by a straight line (representing the decision boundary of a linear classifier). The dashed lines illustrate three failed attempts: each places the feature vector $\featurevec^{(1)}$ (class $\square$) on the same side as at least one feature vector of class $\circ$, so a linear classifier with any of these decision boundaries misclassifies at least one data point. In contrast, the transformed feature vectors $\vz^{(\sampleidx)} = \featuretrafo\big(\featurevec^{(\sampleidx)}\big)$ allow the data points to be separated using a linear classifier
A useful feature transformation $\phi$ often delivers transformed feature vectors in a Hilbert space $\mathcal{H}$ of high (possibly infinite) dimension (Schölkopf and Smola, 2002). Computing and storing the transformed feature vectors $\phi({\mathbf x})$ explicitly can then be infeasible. However, linear methods access the feature vectors of data points only through inner products between pairs of feature vectors (see linear model). For example, the prediction delivered by a trained support vector machine (SVM) is a weighted sum of inner products between the feature vector of a new data point and the feature vectors in the training set. To apply a linear method to the transformed feature vectors, it is therefore enough to know the inner products $\langle \phi({\mathbf x}), \phi({\mathbf x}') \rangle$ for every possible pair of feature vectors ${\mathbf x}, {\mathbf x}' \in \mathcal{X}$ (Boser et al., 1992).

These inner products are the values of a single function of two arguments, the kernel defined by \[ k\big({\mathbf x},{\mathbf x}'\big) := \langle \phi({\mathbf x}), \phi({\mathbf x}') \rangle \text{, for } {\mathbf x}, {\mathbf x}' \in \mathcal{X} \text{.} \] The value $k\big({\mathbf x},{\mathbf x}'\big)$ quantifies the similarity of the feature vectors ${\mathbf x}$ and ${\mathbf x}'$. As an inner product of transformed feature vectors, the kernel is symmetric, and every matrix of pairwise kernel values is positive semi-definite (psd); these two properties characterize kernels. A linear method that is applied to the transformed feature vectors thus requires only the kernel $k$; the feature transformation $\phi$ itself is never evaluated.

Kernel methods reverse this construction: they start from a kernel $k: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ and construct the feature transformation from it. The feature transformation sends a feature vector ${\mathbf x} \in \mathcal{X}$ to the function $k\big({\mathbf x},\cdot\big)$, i.e., the transformed feature vector ${\mathbf z} := \phi({\mathbf x}) = k\big({\mathbf x},\cdot\big)$ is itself a function with domain $\mathcal{X}$, for every ${\mathbf x} \in \mathcal{X}$. These functions belong to the reproducing kernel Hilbert space (RKHS) $\mathcal{H}_{k}$ associated with the kernel $k$ (Aronszajn, 1950).

Take two original feature vectors ${\mathbf x}, {\mathbf x}' \in \mathcal{X}$. The inner product between their transformed feature vectors is a single kernel evaluation, \[ \langle k\big({\mathbf x},\cdot\big), k\big({\mathbf x}',\cdot\big) \rangle = k\big({\mathbf x},{\mathbf x}'\big) \text{.} \] More generally, the inner product between the transformed feature vector $k\big({\mathbf x},\cdot\big)$ and any function $h \in \mathcal{H}_{k}$ is a point evaluation, \[ \langle h, k\big({\mathbf x},\cdot\big) \rangle = h({\mathbf x}) \text{,} \] which is referred to as the reproducing property of the RKHS $\mathcal{H}_{k}$ (Aronszajn, 1950). This reproducing property contains $\langle k\big({\mathbf x},\cdot\big), k\big({\mathbf x}',\cdot\big) \rangle = k\big({\mathbf x},{\mathbf x}'\big)$ as a special case that is obtained for the choice $h = k\big({\mathbf x}',\cdot\big)$. So the conclusion of the forward construction holds here too: the (possibly infinite-dimensional) transformed feature vectors never need to be computed explicitly.

Fig. 2 depicts the RKHS $\mathcal{H}_{k}$ for a training set with two data points: a hypothesis $h$ is a single vector in $\mathcal{H}_{k}$, the two transformed feature vectors span a subspace, and $h$ has an orthogonal projection onto it. The value $\langle h, k\big({\mathbf x},\cdot\big) \rangle = h({\mathbf x})$ is a linear function of the transformed feature vector $k\big({\mathbf x},\cdot\big) \in \mathcal{H}_{k}$, but the function ${\mathbf x} \mapsto h({\mathbf x})$ induced on the original feature space $\mathcal{X}$ is nonlinear in general. Linearity in the original feature vector ${\mathbf x}$ may not even be defined, since the feature space $\mathcal{X}$ need not be a vector space, e.g., when it consists of text documents or graphs.

Figure 2 of the entry kernelmethod
Figure 2: The RKHS $\hilbertspace_{\kernel}$ for a training set with two data points having feature vectors $\featurevec^{(1)}, \featurevec^{(2)}$. Their transformed feature vectors $\kernelmap{\featurevec^{(1)}}{\cdot}$ and $\kernelmap{\featurevec^{(2)}}{\cdot}$ span a two-dimensional subspace (shaded parallelogram). A hypothesis $\hypothesis$ is a vector in $\hilbertspace_{\kernel}$; its prediction for a data point with feature vector $\featurevec$ is the inner product $\innerprod{\hypothesis}{\kernelmap{\featurevec}{\cdot}} = \hypothesis(\featurevec)$. This prediction is a linear function of the transformed feature vector $\kernelmap{\featurevec}{\cdot}$, but the function $\featurevec \mapsto \hypothesis(\featurevec)$ induced on $\featurespace$ is nonlinear in general. The dashed arrow depicts the orthogonal projection $\widetilde{\hypothesis}$ of $\hypothesis$ onto the shaded subspace
Kernel methods can be formulated as regularized empirical risk minimization (RERM) over the RKHS $\mathcal{H}_{k}$, \[ \min_{h \in \mathcal{H}_{k}} \frac{1}{m} \sum_{r=1}^{m} L\left(\big({\mathbf x}^{(r)}, y^{(r)}\big),h \right) + \alpha \left\lVert h \right\rVert_{\mathcal{H}_{k}}^{2} \text{,} \] over a training set with feature vectors ${\mathbf x}^{(r)}$ and labels $y^{(r)}$, for $r = 1, \ldots, m$. The regularization parameter $\alpha > 0$ weights the penalty term $\left\lVert h \right\rVert_{\mathcal{H}_{k}}^{2}$. The loss function $L\left(\big({\mathbf x}^{(r)}, y^{(r)}\big),h \right)$ depends on $h$ only through the prediction $h\big({\mathbf x}^{(r)}\big)$, i.e., it is a function of the prediction $h\big({\mathbf x}^{(r)}\big)$ and the label $y^{(r)}$ only. Since $h \in \mathcal{H}_{k}$, the reproducing property delivers this prediction as an inner product, $h\big({\mathbf x}^{(r)}\big) = \langle h, k\big({\mathbf x}^{(r)},\cdot\big) \rangle$. By the representer theorem, this optimization problem has a minimizer of the form $\widehat{h} = \sum_{r=1}^{m} \beta_{r} k\big({\mathbf x}^{(r)},\cdot\big)$ with expansion coefficients $\beta_{1}, \ldots, \beta_{m} \in \mathbb{R}$ (Schölkopf and Smola, 2002).

Fig. 2 illustrates the underlying projection argument. Replacing a hypothesis $h$ by its orthogonal projection $\widetilde{h}$ onto the subspace spanned by $k\big({\mathbf x}^{(1)},\cdot\big), \ldots, k\big({\mathbf x}^{(m)},\cdot\big)$ leaves the predictions on the training set unchanged and never increases the penalty term. Some minimizer $\widehat{h}$ therefore lies in this subspace and has the above form.

B-normTraining thus reduces to a convex optimization problem in these $m$ coefficients, and the prediction $\widehat{h}({\mathbf x}) = \sum_{r=1}^{m} \beta_{r} k\big({\mathbf x}^{(r)},{\mathbf x}\big)$ requires only kernel evaluations. Examples of such kernel methods (Cristianini and Shawe-Taylor, 2000) are the kernel SVM, obtained for the hinge loss (Hastie et al., 2009, Sect. 12.3.2), and kernel ridge regression, obtained for the squared error loss (Hastie et al., 2009, Sect. 5.8.1). For $\mathcal{X} = \mathbb{R}^{d}$, a widely used choice of the kernel $k$ is the Gaussian kernel $k\big({\mathbf x},{\mathbf x}'\big) = \exp\big(- \left\lVert {\mathbf x} - {\mathbf x}' \right\rVert_{2}^{2} / (2 \sigma^{2}) \big)$ with bandwidth $\sigma > 0$. Kernels are also available for data points without numeric features, such as strings and graphs (Schölkopf and Smola, 2002). Fig. 3 shows the nonlinear decision boundary $\widehat{h}({\mathbf x}) = 0$ of a hypothesis $\widehat{h}$ learned by a Gaussian-kernel method with squared error loss from a training set of two concentric rings that no linear classifier separates. Here, kernel ridge regression fits the labels $y^{(r)} \in \{-1, +1\}$ and the prediction is the sign of the fitted value, a construction referred to as regularized least-squares classification (Rifkin et al., 2003).

Figure 3 of the entry kernelmethod
Figure 3: A Gaussian-kernel method (kernel ridge regression with $\sigma = 0.8$, $\regparam = 10^{-3}$) trained on a training set of two concentric rings. Data points with label $\truelabel = +1$ are drawn as filled circles, those with $\truelabel = -1$ as open squares. The decision boundary $\widehat{\hypothesis}(\featurevec) = 0$ of the learned hypothesis (solid closed curve) encircles the inner ring. The dataset is not linearly separable in the original feature space. Data generated by pythondemos/kernelmethod.py
Kernel methods impose the smoothness assumption through the RKHS norm. That norm $\left\lVert h \right\rVert_{\mathcal{H}_{k}}$ quantifies the smoothness of a hypothesis $h$: it is large for a rapidly varying $h$ and small for a smooth $h$. Moreover, different kernels impose different kinds of smoothness. The Gaussian kernel gives infinitely differentiable functions and penalizes high-frequency components. The Matérn kernels are a family of kernels indexed by a smoothness parameter $\nu > 0$: their RKHS coincides, with equivalent norms, with the Sobolev space of order determined by $\nu$, whose functions have derivatives up to that order in the weak sense (Rasmussen and Williams, 2006, Eq. 4.15; Kanagawa et al., 2018, Example 2.6). The penalty term $\alpha \left\lVert h \right\rVert_{\mathcal{H}_{k}}^{2}$ in the above RERM is the squared RKHS norm. It therefore implements smoothness as quantified by $\left\lVert h \right\rVert_{\mathcal{H}_{k}}$: among hypotheses with the same training error, the RERM objective function is smallest for the hypothesis with the smallest RKHS norm.

B-previewThe choice of kernel determines not only training but also the predictions of the learned hypothesis: for localized kernels, such as the Gaussian kernel, the prediction $\widehat{h}({\mathbf x})$ is dominated by the data points in the training set whose feature vectors are close to ${\mathbf x}$. In this sense, kernel methods implement the smoothness assumption: data points with nearby feature vectors obtain similar predictions. The locality of the predictions also makes kernel methods a soft-weighted analogue of $k$-nearest neighbors ($k$-NN): instead of averaging the labels of a fixed number of nearest data points, the prediction weights all data points in the training set by the kernel value $k\big({\mathbf x}^{(r)},{\mathbf x}\big)$ (Hastie et al., 2009, Sect. 6.1).

Synonyms: kernel machine.

See also: kernel, feature transformation, Hilbert space, reproducing kernel Hilbert space, support vector machine, kernel ridge regression, smoothness assumption.

References

  1. Lampert (2009). Kernel Methods in Computer Vision. Found. Trends Comput. Graph. Vis.. doi.org/10.1561/0600000027
  2. Cristianini and Shawe-Taylor (2000). An Introduction to Support Vector Machines and Other Kernel-based Learning Methods. Cambridge Univ. Press. doi.org/10.1017/cbo9780511801389
  3. Schölkopf and Smola (2002). Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press. doi.org/10.7551/mitpress/4175.001.0001
  4. Boser et al. (1992). A Training Algorithm for Optimal Margin Classifiers. Proc. 5th Annual Workshop on Computational Learning Theory (COLT). doi.org/10.1145/130385.130401
  5. Aronszajn (1950). Theory of Reproducing Kernels. Trans. Am. Math. Soc.. doi.org/10.1090/S0002-9947-1950-0051437-7
  6. Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media. doi.org/10.1007/978-0-387-84858-7
  7. Rifkin et al. (2003). Regularized Least-Squares Classification. Advances in Learning Theory: Methods, Model and Applications. doi.org/10.21236/ada454981
  8. Rasmussen and Williams (2006). Gaussian Processes for Machine Learning. MIT Press. doi.org/10.7551/mitpress/3206.001.0001
  9. Kanagawa et al. (2018). Gaussian Processes and Kernel Methods: A Review on Connections and Equivalences. arxiv.org/abs/1807.02582

Cite this entry

Published version: doi.org/10.1007/978-981-92-2012-0_16-1, in Dictionary of Applied Machine Learning (Springer Nature Singapore). Please cite that version. This page corresponds to version 1 of that entry.

@incollection{dictml_kernelmethod,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {kernel method},
  booktitle = {Dictionary of Applied Machine Learning},
  publisher = {Springer Nature Singapore},
  year = {2026},
  doi = {10.1007/978-981-92-2012-0_16-1},
  isbn = {978-981-92-2012-0},
  url = {https://doi.org/10.1007/978-981-92-2012-0_16-1}
}