Dictionary of Applied Machine Learning: 114 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

support vector machine (SVM)

Updated on 2026-10-29

Readiness 100/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Markdown Cite this entry

See also binary classification decision boundary kernel method

The support vector machine (SVM) is a binary classification method that learns a linear classifier by regularized empirical risk minimization (RERM), combining the hinge loss with a squared-norm penalty term. For a linearly separable training set and sufficiently weak regularization, the solution is the maximum-margin separating hyperplane. This hyperplane is fully determined by the feature vectors closest to it, which are called support vectors. The kernel SVM is obtained by combining the basic SVM with a feature transformation derived from a kernel function.

Figure produced by the Python demo of the entry support vector machine (SVM)
Figure written by the entry's Python demo svm.py, which recomputes what the entry states: the script block by block · open in Colab

Definition

A handwritten digit is recognized from the pixel values of its image, and a text document is sorted by topic from the frequencies of its words (Cristianini and Shawe-Taylor, 2000). In both tasks a numeric feature vector has to be mapped to a label, and the SVM has been applied to both. The SVM is a binary classification method for data points with feature space $\mathcal{X}=\mathbb{R}^{d}$. In its simplest form, the SVM learns a linear classifier with decision boundary \[\{ {\mathbf x} \in \mathbb{R}^{d} : {\mathbf w}^{\top} {\mathbf x} + b = 0 \}.\] The linear classifier is parameterized by a nonzero vector ${\mathbf w} \in \mathbb{R}^{d} \setminus \{\mathbf{0}\}$ and an offset $b \in \mathbb{R}$. The vector ${\mathbf w}$ is the normal vector to the decision boundary and the offset $b$ shifts the decision boundary away from the origin. For a data point with feature vector ${\mathbf x}$ and label $y \in \{-1, +1\}$, the SVM delivers the prediction $\widehat{y} = \operatorname{sign}({\mathbf w}^{\top} {\mathbf x} + b)$.

P-nonsepThe SVM can be formulated as an instance of regularized empirical risk minimization (RERM) for learning a linear classifier from a training set that consists of $m$ data points. The model parameters $(\widehat{{\mathbf w}}, \widehat{b})$ of the linear classifier are obtained as \begin{equation} \label{eq:svm_rerm} (\widehat{{\mathbf w}}, \widehat{b}) = \mathop{\mathrm{arg\,min}}_{{\mathbf w} \in \mathbb{R}^{d},\, b \in \mathbb{R}} \underbrace{\frac{1}{m} \sum_{r=1}^{m} \max\{0,\, 1 - y^{(r)} ({\mathbf w}^{\top} {\mathbf x}^{(r)} + b)\} + \alpha \left\lVert {\mathbf w} \right\rVert_{2}^{2}}_{:= f({\mathbf w}, b)}\text{.} \end{equation} The SVM objective function consists of the average hinge loss on the training set plus a penalty term $\alpha \left\lVert {\mathbf w} \right\rVert_{2}^{2}$ with regularization parameter $\alpha > 0$ (Hastie et al., 2009, Sect. 12.3.2, p. 426; Shalev-Shwartz and Ben-David, 2014, Sect. 15.2, Eq. (15.5)). The penalty term in the SVM problem \(\eqref{eq:svm_rerm}\) is the scaled squared Euclidean norm $\alpha \left\lVert {\mathbf w} \right\rVert_{2}^{2}$, the same penalty term as in ridge regression. It does not involve the model parameter $b$, which is therefore left unpenalized.

In linear regression, the penalty term estimates how much higher the loss is on data points outside the training set than on it (Hastie et al., 2009, Ch. 7; see linear regression), an interpretation that requires the quadratic structure of the squared error loss. For the SVM, the penalty term instead controls the generalization gap: the hinge loss is Lipschitz continuous in ${\mathbf w}$, with a constant given by the Euclidean norm of the feature vector. Adding the penalty term $\alpha \left\lVert {\mathbf w} \right\rVert_{2}^{2}$ to the average hinge loss has three effects: First, it makes the overall objective function in the SVM problem \(\eqref{eq:svm_rerm}\) strongly convex in ${\mathbf w}$, so that its solution $\widehat{{\mathbf w}}$ is unique. Second, it controls how much that solution varies if the data points in the training set change. Third, it keeps $\left\lVert {\mathbf w} \right\rVert_{2}$ small and therefore the margins wide (Shalev-Shwartz and Ben-David, 2014, Sect. 15.2).

If the feature vectors in the training set have bounded Euclidean norm, the expectation of the difference between the risk and the empirical hinge loss of the learned linear classifier decreases in proportion to $1/(\alpha m)$ (Shalev-Shwartz and Ben-David, 2014, Cor. 13.6, Cor. 15.7).

P-svThe solution of the SVM problem \(\eqref{eq:svm_rerm}\) has a geometric meaning when the training set is linearly separable, i.e., there is some choice for $({\mathbf w}, b)$ such that $\operatorname{sign}({\mathbf w}^{\top} {\mathbf x}^{(r)} + b) y^{(r)}=1$ for $r=1, \ldots, m$. For a sufficiently small $\alpha$, the solution of the SVM problem \(\eqref{eq:svm_rerm}\) determines the separating hyperplane that is farthest from the feature vectors in the training set (see Fig. 1). The distance between this hyperplane and the nearest feature vectors, which are precisely the support vectors discussed below, is referred to as the margin and is given by $1/\left\lVert \widehat{{\mathbf w}} \right\rVert_{2}$ (Boser et al., 1992; Cortes and Vapnik, 1995; Cristianini and Shawe-Taylor, 2000).

Figure 1 of the entry svm
Figure 1: Maximum-margin separation for a linearly separable training set. The decision boundary $\widehat{\weights}^{\top}\featurevec + \widehat{\offset} = 0$ (solid line) lies halfway between two parallel hyperplanes $\widehat{\weights}^{\top}\featurevec + \widehat{\offset} = \pm 1$ (dashed lines). The support vectors (open square and filled circle inside red rings) are the feature vectors that lie on these hyperplanes. The margin $\gamma = 1/\normgeneric{\widehat{\weights}}{2}$ is the distance from the decision boundary to the support vectors. Data points with label $\truelabel=+1$ are drawn as filled circles, those with $\truelabel=-1$ as open squares. The dashed circle of radius $\gamma$ shows that any support vector can be displaced by a perturbation with Euclidean norm less than $\gamma$ without crossing the decision boundary
The margin of the learned hyperplane measures robustness against perturbations of the features of a data point. Perturbing a feature vector ${\mathbf x}$ by a vector $\boldsymbol{\delta}$ changes the score $\widehat{{\mathbf w}}^{\top}{\mathbf x} + \widehat{b}$ by $\widehat{{\mathbf w}}^{\top}\boldsymbol{\delta}$, whose magnitude is at most $\left\lVert \widehat{{\mathbf w}} \right\rVert_{2} \left\lVert \boldsymbol{\delta} \right\rVert_{2}$ by the Cauchy-Schwarz inequality. A data point therefore stays correctly classified under every feature perturbation with $\left\lVert \boldsymbol{\delta} \right\rVert_{2} < \gamma$, as illustrated by the dashed circle around a support vector in Fig. 1. Data points whose feature vectors lie farther from the decision boundary tolerate even larger perturbations: the tolerated radius equals the distance of the feature vector from the decision boundary, which is at least $\gamma$ and equals $\gamma$ precisely for the support vectors. Maximizing the margin maximizes this worst-case tolerated radius, so the maximum-margin classifier is the separating classifier that is robust against the largest feature perturbations. The same holds beyond the separable case: the SVM problem \(\eqref{eq:svm_rerm}\) is equivalent to minimizing the hinge loss under the worst case over bounded perturbations of the feature vectors (Xu et al., 2009, Thm. 3).

P-violEvery solution of the SVM problem \(\eqref{eq:svm_rerm}\) has a structure that holds whether or not the training set is linearly separable. In particular, \[\widehat{{\mathbf w}} = \sum_{r=1}^{m} \beta_{r} y^{(r)} {\mathbf x}^{(r)}\] with expansion coefficients $0 \leq \beta_{r} \leq 1/(2 \alpha m)$ that are nonzero only for the feature vectors with \[y^{(r)} (\widehat{{\mathbf w}}^{\top} {\mathbf x}^{(r)} + \widehat{b}) \leq 1.\] These feature vectors are referred to as support vectors since they entirely determine the SVM solution $(\widehat{{\mathbf w}}, \widehat{b})$ (Boser et al., 1992; Cortes and Vapnik, 1995, Sect. 2). Applying a perturbation to any feature vector that is not a support vector leaves $\widehat{{\mathbf w}}$, $\widehat{b}$ unchanged, provided the perturbation is small enough to preserve the strict inequality $y (\widehat{{\mathbf w}}^{\top} {\mathbf x} + \widehat{b}) > 1$ (see Fig. 2, whose training set contains an outlier).

Figure 2 of the entry svm
Figure 2: Non-separable training set of $\samplesize = 7$ data points in feature space $\reals^{2}$. The solution of the SVM problem \(\eqref{eq:svm_rerm}\) places the decision boundary $\widehat{\weights}^{\top}\featurevec + \widehat{\offset} = 0$ (solid line) between the two hyperplanes $\widehat{\weights}^{\top}\featurevec + \widehat{\offset} = \pm 1$ (dashed lines) at distance $\gamma = 1/\normgeneric{\widehat{\weights}}{2}$. These two hyperplanes contain four of the support vectors (red rings), given by the data points with margin equal to $\gamma$. The outlier, labeled $+1$, is also a support vector; it is misclassified beyond the opposite margin: $\truelabel(\widehat{\weights}^{\top}\featurevec + \widehat{\offset}) = -2 < -1$, with hinge loss value $\xi$. The two data points (open square and filled circle not inside red rings) farther from the decision boundary than the margin $\gamma$ are not support vectors. Data generated by pythondemos/svm.py
Like logistic regression and linear regression, the SVM learns the model parameters $({\mathbf w}, b)$ of a linear model. In contrast to logistic regression and linear regression, the SVM objective function $f({\mathbf w}, b)$ is non-smooth because of the hinge loss. However, since the SVM objective function is convex, the optimization problem \(\eqref{eq:svm_rerm}\) can be solved by convex optimization methods. One such method is subgradient descent, which is obtained from gradient descent (GD) by replacing the gradient of the objective function with a subgradient (Shor, 1985; Bertsekas, 2016; Boyd and Vandenberghe, 2004).

Starting from an initial choice ${\mathbf w}^{(0)}, b^{(0)}$ of the model parameters, subgradient descent repeatedly applies the update \[ \big({\mathbf w}^{(t+1)}, b^{(t+1)}\big) = \big({\mathbf w}^{(t)}, b^{(t)}\big) - \eta^{(t)} {\mathbf g}^{(t)} \text{, for } t = 0,1,\ldots \text{,} \] with a step size $\eta^{(t)} > 0$ and a subgradient ${\mathbf g}^{(t)} \in \partial f\big({\mathbf w}^{(t)}, b^{(t)}\big)$ of the SVM objective function \(\eqref{eq:svm_rerm}\). Inserting the SVM objective function \(\eqref{eq:svm_rerm}\) into the generic update yields the explicit update \[ \begin{aligned} {\mathbf w}^{(t+1)} &= \big(1 - 2 \eta^{(t)} \alpha\big) {\mathbf w}^{(t)} + \frac{\eta^{(t)}}{m} \sum_{r=1}^{m} \beta^{(t)}_{r} y^{(r)} {\mathbf x}^{(r)} \text{,} \\ b^{(t+1)} &= b^{(t)} + \frac{\eta^{(t)}}{m} \sum_{r=1}^{m} \beta^{(t)}_{r} y^{(r)} \text{,} \end{aligned} \] with expansion coefficients $\beta^{(t)}_{r} \in [0, 1]$, for $r = 1, \ldots, m$. The coefficient $\beta^{(t)}_{r}$ is nonzero only for data points that satisfy $y^{(r)} \big( \big({\mathbf w}^{(t)}\big)^{\top} {\mathbf x}^{(r)} + b^{(t)} \big) \leq 1$, i.e., data points on which the hinge loss is nonzero or non-differentiable at the current model parameters. Each iteration scales ${\mathbf w}^{(t)}$ by the factor $1 - 2 \eta^{(t)} \alpha$ (the effect of the penalty term) and adds corrections $y^{(r)} {\mathbf x}^{(r)}$ only from these data points.

In particular, for the initialization ${\mathbf w}^{(0)} = \mathbf{0}$, every iterate ${\mathbf w}^{(t)}$ is a weighted sum $\sum_{r=1}^{m} \beta_{r} y^{(r)} {\mathbf x}^{(r)}$ of the feature vectors in the training set. This is the expansion of the solution $\widehat{{\mathbf w}}$ in the support vectors stated above, built up one iteration at a time: the coefficients of the iterate are nonzero only for the data points satisfying $y^{(r)} \big( \widehat{{\mathbf w}}^{\top} {\mathbf x}^{(r)} + \widehat{b} \big) \leq 1$. For a suitably diminishing step size, e.g., $\eta^{(t)} = 1/(t+1)$, the iterates $\big({\mathbf w}^{(t)}, b^{(t)}\big)$ converge to a solution of the SVM problem \(\eqref{eq:svm_rerm}\) (Shor, 1985).

P-alphaAn overly large regularization parameter $\alpha$ makes the SVM underfit: the above expansion of $\widehat{{\mathbf w}}$ implies $\left\lVert \widehat{{\mathbf w}} \right\rVert_{2} \leq \rho/(2\alpha)$, with $\rho$ the largest Euclidean norm of a feature vector in the training set, so $\widehat{{\mathbf w}}$ shrinks toward $\mathbf{0}$. Consider a training set with unequal class sizes. For sufficiently large $\alpha$, the predictions approach the constant majority-class prediction $\operatorname{sign}(\widehat{b})$ and every data point of the minority class is a misclassified support vector. The training set of Fig. 2 also has unequal class sizes, four data points labeled $y = +1$ and three labeled $y = -1$, but with the value $\alpha = 1/14$ used there, the SVM misclassifies one data point that can be considered an outlier. Increasing the value of $\alpha$ to $50$ for that training set shrinks $\left\lVert \widehat{{\mathbf w}} \right\rVert_{2}$, makes every prediction on the training set equal to $+1$, and misclassifies all three data points of the minority class.

P-kernelThe basic SVM \(\eqref{eq:svm_rerm}\) learns a linear classifier on the feature space $\mathcal{X} = \mathbb{R}^{d}$. It can be generalized to a classification method for data points whose feature vectors lie in an arbitrary feature space $\mathcal{X}$ by first applying a feature transformation $\phi: \mathcal{X} \to \mathcal{X}'$ with $\mathcal{X}' = \mathbb{R}^{d}$ and then learning a linear classifier on the transformed feature vectors. By choosing $\phi$ (and $d$) appropriately, any given training set with distinct feature vectors can be made linearly separable in $\mathcal{X}'$, e.g., by mapping each distinct feature vector to its own coordinate axis (Cristianini and Shawe-Taylor, 2000). A general construction of a feature transformation $\phi$ is via a kernel $k: \mathcal{X} \times \mathcal{X} \to \mathbb{R}$. In this construction, the transformed feature space is, in general, not $\mathbb{R}^{d}$ but a Hilbert space $\mathcal{H}$, which can be infinite-dimensional (see kernel method). The resulting method is then referred to as a kernel SVM (Schölkopf and Smola, 2002).

Synonyms: maximum-margin classifier.

See also: binary classification, hinge loss, margin, hyperplane, decision boundary, robustness, kernel method, subgradient descent.

References

  1. Cristianini and Shawe-Taylor (2000). An Introduction to Support Vector Machines and Other Kernel-based Learning Methods. Cambridge Univ. Press. doi.org/10.1017/cbo9780511801389
  2. Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media. doi.org/10.1007/978-0-387-84858-7
  3. Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. Cambridge Univ. Press. doi.org/10.1017/cbo9781107298019
  4. Boser et al. (1992). A Training Algorithm for Optimal Margin Classifiers. Proc. 5th Annual Workshop on Computational Learning Theory (COLT). doi.org/10.1145/130385.130401
  5. Cortes and Vapnik (1995). Support-vector networks. Machine learning. doi.org/10.1007/BF00994018
  6. Xu et al. (2009). Robustness and Regularization of Support Vector Machines. J. Mach. Learn. Res.. www.jmlr.org/papers/v10/xu09b.html
  7. Shor (1985). Minimization Methods for Non-differentiable Functions. Springer. doi.org/10.1007/978-3-642-82118-9
  8. Bertsekas (2016). Nonlinear Programming. Athena Scientific.
  9. Boyd and Vandenberghe (2004). Convex Optimization. Cambridge Univ. Press. doi.org/10.1017/CBO9780511804441
  10. Schölkopf and Smola (2002). Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press. doi.org/10.7551/mitpress/4175.001.0001

Cite this entry

@misc{dictml_svm,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {support vector machine (SVM)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-04},
  url = {https://dictionaryofml.org/terms/svm.html}
}