Dictionary of Applied Machine Learning: 115 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

norm

Updated on 2026-11-08

Readiness 100/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Markdown Cite this entry

See also Hilbert space inner product linear regression

A norm on a vector space is a function that assigns each vector a nonnegative number, its length. A norm is definite and homogeneous, and it satisfies the triangle inequality. Every norm defines a metric via the norm of the difference of two vectors, and every inner product induces a norm, but not every norm is induced by one. The $\ell_{p}$-norms on $\mathbb{R}^{d}$ include the $\ell_{1}$-norm, the Euclidean norm, and the $\ell_{\infty}$-norm; among them only the Euclidean norm is induced by an inner product. In machine learning (ML), norms are used to construct loss functions and regularizers.

Figure produced by the Python demo of the entry norm
Figure written by the entry's Python demo norm.py, which recomputes what the entry states: the script block by block · open in Colab

Definition

P-axiomsThe hourly temperature recordings of one day at a weather station form the feature vector ${\mathbf x} \in \mathbb{R}^{d}$ of that day, with one entry per hour. Many machine learning (ML) methods use a measure for the size of such a vector or for the distance between two of them. A norm provides both: it assigns each vector a length, and the length of the difference of two vectors is their distance.

P-metricFormally, a norm $\left\lVert \, \cdot \, \right\rVert_{}$ on a vector space $\mathcal{V}$ over the field $\mathbb{R}$ is a function $\left\lVert \, \cdot \, \right\rVert_{}: \mathcal{V} \to \mathbb{R}_{+}$ that satisfies the conditions below (Horn and Johnson, 2013). These conditions hold for all vectors ${\mathbf u}, {\mathbf v} \in \mathcal{V}$ and all scalars $\beta \in \mathbb{R}$, with $|\beta|$ the absolute value of $\beta$:

The same conditions define a norm on a vector space over the field of complex numbers $\mathbb{C}$, with $\beta \in \mathbb{C}$ and $|\beta|$ its modulus. A function that satisfies homogeneity and the triangle inequality but assigns the value zero to some nonzero vector is a seminorm, not a norm: on $\mathbb{R}^{2}$, the map ${\mathbf u} = (u_{1}, u_{2})^{\top} \mapsto |u_{1}|$, which keeps only the first entry, is a seminorm that is zero on $(0,1)^{\top}$.

P-innerEvery norm defines a metric by $d({\mathbf u},{\mathbf v}) := \left\lVert {\mathbf u} - {\mathbf v} \right\rVert_{}$, making $\left( \mathcal{V},d(\cdot,\cdot) \right)$ a metric space. Some norms are induced by an inner product $\langle \cdot, \cdot \rangle$, a function of two vectors that is symmetric, linear in each argument, and positive-definite (on $\mathbb{R}^{d}$, $\langle {\mathbf u}, {\mathbf v} \rangle = {\mathbf u}^{\top} {\mathbf v}$). On an inner product space, $\left\lVert {\mathbf u} \right\rVert_{} := \sqrt{\langle {\mathbf u}, {\mathbf u} \rangle}$ is a norm (Axler, 2024, Thm. 6.14 and Thm. 6.17), and the norm of a Hilbert space is of this form. An induced norm satisfies the parallelogram identity $\left\lVert {\mathbf u} + {\mathbf v} \right\rVert_{}^{2} + \left\lVert {\mathbf u} - {\mathbf v} \right\rVert_{}^{2} = 2 \left\lVert {\mathbf u} \right\rVert_{}^{2} + 2 \left\lVert {\mathbf v} \right\rVert_{}^{2}$, obtained by expanding $\langle {\mathbf u} \pm {\mathbf v}, {\mathbf u} \pm {\mathbf v} \rangle$.

P-lpFor the vector space $\mathcal{V} = \mathbb{R}^{d}$ of feature vectors ${\mathbf x} = (x_{1}, \ldots, x_{d})^{\top}$ and a number $p \geq 1$, the $\ell_p$-norm of ${\mathbf x}$ is defined as $\left\lVert {\mathbf x} \right\rVert_{p} = \left( \displaystyle \sum_{j=1}^{d} |x_{j}|^p \right)^{1/p}$. It satisfies the triangle inequality by Minkowski's inequality and is therefore a norm (Horn and Johnson, 2013, Sect. 5.2, Eq. (5.2.4)). Its instances include the $\ell_1$-norm $\left\lVert {\mathbf x} \right\rVert_{1} = \sum_{j=1}^{d} |x_{j}|$ and the $\ell_2$-norm $\left\lVert {\mathbf x} \right\rVert_{2} = \big( \sum_{j=1}^{d} x_{j}^{2} \big)^{1/2}$ (the Euclidean norm); the limit $p \to \infty$ yields the $\ell_\infty$-norm $\left\lVert {\mathbf x} \right\rVert_{\infty} = \max_{j = 1, \ldots, d} |x_{j}|$. The $\ell_2$-norm is induced by the inner product ${\mathbf u}^{\top} {\mathbf v}$, and it is the only $\ell_p$-norm induced by an inner product. For ${\mathbf u} = (1, 0)^{\top}$ and ${\mathbf v} = (0, 1)^{\top}$ in $\mathbb{R}^{2}$, both ${\mathbf u} + {\mathbf v}$ and ${\mathbf u} - {\mathbf v}$ have $\ell_p$-norm $2^{1/p}$, so the left-hand side of the parallelogram identity is $2 \cdot 2^{2/p}$ while the right-hand side is $4$, and the two agree only for $p = 2$. The left-hand side equals $8$ for the $\ell_1$-norm and $2$ for the $\ell_\infty$-norm. The three norms differ in which vectors they count as having unit length. The unit sphere $\mathbb{S}^{(1)} := \big\{ {\mathbf u} \in \mathbb{R}^{2} : \left\lVert {\mathbf u} \right\rVert_{} = 1 \big\}$ of a norm on $\mathbb{R}^{2}$ contains those vectors, and Fig. 1 shows it for the $\ell_1$-, the $\ell_2$-, and the $\ell_\infty$-norm.

Figure 1 of the entry norm
Figure 1: Unit spheres $\sphere{1}$ of the $\ell_p$-norms on $\reals^{2}$, for vectors $\vu = (u_{1}, u_{2})^{\top}$ and $p=1$ (dashed diamond), $p=2$ (solid circle), and $p=\infty$ (dotted square)
ML methods use norms to construct loss functions and regularizers. In linear regression, the label vector ${\mathbf y} \in \mathbb{R}^{m}$ stacks the labels of a training set of size $m$ and the feature matrix ${\mathbf X} \in \mathbb{R}^{m \times d}$ stacks its feature vectors. The average squared error loss of a linear hypothesis with model parameters ${\mathbf w}$ is then the scaled squared Euclidean norm $(1/m) \left\lVert {\mathbf y} - {\mathbf X} {\mathbf w} \right\rVert_{2}^{2}$ of the residual ${\mathbf y} - {\mathbf X} {\mathbf w}$. Ridge regression adds the penalty term $\alpha \left\lVert {\mathbf w} \right\rVert_{2}^{2}$ to this training error, with a hyperparameter $\alpha > 0$, while the least absolute shrinkage and selection operator (Lasso) adds $\alpha \left\lVert {\mathbf w} \right\rVert_{1}$ (Tibshirani, 1996).

P-mlThe choice of the norm determines the learned hypothesis by empirical risk minimization (ERM). Consider the two optimization problems \[ \min_{{\mathbf w} \in \mathbb{R}^{d}} \left\lVert {\mathbf y} - {\mathbf X} {\mathbf w} \right\rVert_{2} \quad \text{ and } \quad \min_{{\mathbf w} \in \mathbb{R}^{d}} \left\lVert {\mathbf y} - {\mathbf X} {\mathbf w} \right\rVert_{1} \text{,} \] which differ only in the norm of the residual. Squaring the $\ell_2$-norm does not change the set of minimizers, so minimizing the $\ell_2$-norm of the residual is equivalent to minimizing the average squared error loss. The distinction between induced and other norms matters here. An induced norm defines angles and orthogonal projections, and the squared $\ell_2$-norm of the residual is a quadratic function of ${\mathbf w}$, so the least squares problem has a closed-form solution, obtained from the normal equations (Boyd and Vandenberghe, 2004, Sect. 1.2). This is why the $\ell_2$-norm is the norm of the loss function of linear regression. Minimizing the $\ell_1$-norm of the residual is a linear program (Boyd and Vandenberghe, 2004, Sect. 6.1.1), which has no closed-form solution and is solved iteratively, as is the Lasso problem with its $\ell_1$-norm penalty.

P-fitFig. 2 shows a training set of four data points, one of which is an outlier, and two linear hypotheses $h(x) = w x$ learned from it. Minimizing the squared $\ell_2$-norm of the residual gives the slope $\widehat{w} = 46/30$. This solution is moved toward the outlier because squaring makes the largest residual the main part of the training error. Minimizing the $\ell_1$-norm yields $\widehat{w} = 1$, which predicts the three remaining data points exactly and leaves a single large residual at the outlier. The hypothesis learned with the $\ell_1$-norm is robust to the outlier in the sense that one unusual data point barely moves it (Boyd and Vandenberghe, 2004, Sect. 6.1.2), while the one learned by minimizing the squared $\ell_2$-norm is not (see robustness). Both slopes are learned from the training set; which of the two hypotheses has the smaller risk on data points outside it depends on whether outliers are present there as well.

Figure 2 of the entry norm
Figure 2: Two linear hypotheses $\hypothesis(\feature) = \weight \feature$ learned from the same training set of four data points (filled circles) by minimizing a norm of the residual. The squared $\ell_2$-norm yields the solid line with slope $\widehat{\weight} = 46/30$, moved toward the outlier; the $\ell_1$-norm yields the dashed line with slope $\widehat{\weight} = 1$, which predicts the three remaining data points exactly and leaves one large residual (dotted vertical segment) at the outlier. Data generated by pythondemos/norm.py

See also: vector space, metric, metric space, Euclidean space, Hilbert space, inner product, Euclidean norm, regularizer, linear regression, least absolute shrinkage and selection operator.

References

  1. Horn and Johnson (2013). Matrix Analysis. Cambridge Univ. Press. doi.org/10.1017/cbo9781139020411
  2. Axler (2024). Linear Algebra Done Right. Springer. doi.org/10.1007/978-3-031-41026-0
  3. Tibshirani (1996). Regression Shrinkage and Selection via the Lasso. J. Roy. Statist. Soc.: Ser. B (Methodological). doi.org/10.1111/j.2517-6161.1996.tb02080.x
  4. Boyd and Vandenberghe (2004). Convex Optimization. Cambridge Univ. Press. doi.org/10.1017/CBO9780511804441

Cite this entry

@misc{dictml_norm,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {norm},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/norm.html}
}