Dictionary of Applied Machine Learning

norm

Updated on 2026-08-31

Typeset PDF version — the authoritative form of this entry

Python demo — a script that recomputes what this entry states and prints one line per check

A norm on a vector space is a function that assigns each vector a nonnegative number, its length. A norm is definite and homogeneous, and it satisfies the triangle inequality. Every norm defines a metric via the norm of the difference of two vectors, and every inner product induces a norm. The $\ell_{p}$-norms on $\reals^{\featuredim}$ include the $\ell_{1}$-norm, the Euclidean norm, and the $\ell_{\infty}$-norm. In machine learning (ML), norms are used to construct loss functions and regularizers.

Definition

P-axiomsConsider a data point that is represented by a feature vector $\featurevec \in \reals^{\featuredim}$, e.g., the hourly temperature recordings of one day at a weather station. Many machine learning (ML) methods use a measure for the size of such a vector or for the distance between two of them. A norm provides this measure: a norm $\normgeneric{\, \cdot \,}{}$ on a vector space $\vecspace$ over the field $\reals$ is a function $\normgeneric{\, \cdot \,}{}: \vecspace \to \reals_{+}$ that satisfies the following conditions (Horn and Johnson, 2013) for all $\vu, \vv \in \vecspace$ and $\expcoeff \in \reals$, with $|\expcoeff|$ the absolute value:

The same conditions define a norm on a vector space over the complex numbers, with $|\expcoeff|$ the modulus. A function that satisfies homogeneity and the triangle inequality but assigns the value zero to some non-zero vector is a seminorm, not a norm: on $\reals^{2}$, the map $\vu \mapsto |u_{1}|$ is a seminorm that vanishes on $(0,1)^{\top}$.

P-metricEvery norm defines a metric by $\metric{\vu}{\vv} \defeq \normgeneric{\vu - \vv}{}$, making $\pair{\vecspace}{\metric{\cdot}{\cdot}}$ a metric space.

P-innerOn an inner product space, the inner product induces a norm via $\normgeneric{\vu}{} \defeq \sqrt{\innerprod{\vu}{\vu}}$; the norm of a Hilbert space is of this form.

P-lpFor $1 \leq p < \infty$, the $\ell_p$-norm of a vector $\featurevec \in \reals^{\featuredim}$ is defined as $\normgeneric{\featurevec}{p} = \left( \displaystyle \sum_{\featureidx=1}^{\featuredim} |\feature_{\featureidx}|^p \right)^{1/p}$. Its instances include the $\ell_1$-norm and the $\ell_2$-norm (or Euclidean norm); the limit $p \to \infty$ yields the $\ell_\infty$-norm $\normgeneric{\featurevec}{\infty} = \max_{\featureidx = 1, \ldots, \featuredim} |\feature_{\featureidx}|$. Not every norm is induced by an inner product. For an induced norm, expanding $\innerprod{\vu \pm \vv}{\vu \pm \vv}$ yields the parallelogram identity $\normgeneric{\vu + \vv}{}^{2} + \normgeneric{\vu - \vv}{}^{2} = 2 \normgeneric{\vu}{}^{2} + 2 \normgeneric{\vv}{}^{2}$. The $\ell_1$-norm on $\reals^{2}$ violates this identity for $\vu = (1, 0)^{\top}$ and $\vv = (0, 1)^{\top}$: the left-hand side equals $8$, the right-hand side $4$. The unit sphere $\sphere{1} \defeq \big\{ \vu \in \reals^{2} : \normgeneric{\vu}{} = 1 \big\}$ of a norm on $\reals^{2}$ consists of the vectors of unit length; Fig. 1 shows the unit spheres of these three norms.

P-mlML methods use norms to construct loss functions and regularizers. In linear regression, the average squared error loss of a linear hypothesis with model parameters $\weights$ on a training set of size $\samplesize$ is the scaled squared Euclidean norm $(1/\samplesize) \normgeneric{\labelvec - \featuremtx \weights}{2}^{2}$ of the residual $\labelvec - \featuremtx \weights$. Ridge regression adds the penalty term $\regparam \normgeneric{\weights}{2}^{2}$ to this training error, while the least absolute shrinkage and selection operator (Lasso) adds $\regparam \normgeneric{\weights}{1}$ (Tibshirani, 1996).

P-fitThe choice of the norm determines the learned hypothesis by empirical risk minimization (ERM). Consider the two optimization problems \begin{equation} \min_{\weights \in \reals^{\featuredim}} \normgeneric{\labelvec - \featuremtx \weights}{2} \quad \text{ and } \quad \min_{\weights \in \reals^{\featuredim}} \normgeneric{\labelvec - \featuremtx \weights}{1} \text{,} \end{equation} which differ only in the norm of the residual. Squaring the $\ell_2$-norm does not change the set of minimizers, so minimizing the $\ell_2$-norm of the residual is equivalent to minimizing the average squared error loss. Fig. 2 shows a training set of four data points, one of which is an outlier, and two linear hypotheses $\hypothesis(\feature) = \weight \feature$ learned from it. Minimizing the squared $\ell_2$-norm of the residual yields the slope $\widehat{\weight} = 46/30$. This solution is pulled toward the outlier because squaring makes the largest residual dominate the training error. Minimizing the $\ell_1$-norm yields $\widehat{\weight} = 1$, which fits the three remaining data points exactly and leaves a single large residual at the outlier.

Figure 1 of the entry norm
Figure 1: Unit spheres $\sphere{1}$ of the $\ell_p$-norms on $\reals^{2}$ for $p=1$ (dashed diamond), $p=2$ (solid circle), and $p=\infty$ (dotted square)
Figure 2 of the entry norm
Figure 2: Two linear hypotheses $\hypothesis(\feature) = \weight \feature$ learned from the same training set of four data points by minimizing a norm of the residual. The squared $\ell_2$-norm yields the solid line, pulled toward the outlier; the $\ell_1$-norm yields the dashed line, which fits the three remaining data points exactly and leaves one large residual (dotted) at the outlier. Data generated by pythondemos/norm.py
See also: vector space, metric, metric space, Euclidean space, Hilbert space, inner product, Euclidean norm, regularizer, linear regression, Lasso.

References

  1. Horn and Johnson (2013). Matrix Analysis. Cambridge Univ. Press. doi.org/10.1017/cbo9781139020411
  2. Tibshirani (1996). Regression Shrinkage and Selection via the Lasso. J. Roy. Statist. Soc.: Ser. B (Methodological). doi.org/10.1111/j.2517-6161.1996.tb02080.x

Cite this entry

@misc{dictml_norm,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {norm},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-09-22},
  url = {https://dictionaryofml.org/terms/norm.html}
}