inner product

Typeset PDF version — the authoritative form of this entry

Many machine learning (ML) applications involve data points whose features form numeric arrays. Examples include the color intensities of image pixels and the amplitudes at regular intervals of a sensor signal. Another example is the embeddings of tokens used by large language models (LLMs). These numeric arrays can be naturally represented by vectors in some vector space. Many ML methods rely on measuring the similarity between vectors that represent data points. Each neuron of an artificial neural network (ANN) matches its input against a learned template. The attention unit of an LLM compares the query vector of a token with the key vector of another token. An inner product is a precise mathematical formalization of a notion of similarity between two vectors in a vector space.

Definition

Consider a vector space $\vecspace$ over the field of real numbers $\reals$. An inner product in $\vecspace$ is a function \[ \innerprod{\cdot}{\cdot}: \vecspace \times \vecspace \to \reals \] that satisfies the following properties for all $\vu, \vv, \vw \in \vecspace$ and all scalars $\expcoeff \in \reals$ (Axler, 2015, Ch. 6):

Symmetry: $\innerprod{\vu}{\vv} = \innerprod{\vv}{\vu}$;
Linearity in the first argument: $\innerprod{\expcoeff \vu + \vw}{\vv} = \expcoeff \innerprod{\vu}{\vv} + \innerprod{\vw}{\vv}$;
Positive-definiteness: $\innerprod{\vu}{\vu} \geq 0$, with equality if and only if $\vu = \mathbf{0}$.

For a vector space over the field of complex numbers $\mathbb{C}$, symmetry is replaced by conjugate symmetry, $\innerprod{\vu}{\vv} = \overline{\innerprod{\vv}{\vu}}$, while linearity in the first argument is retained; the inner product is then conjugate-linear in the second argument (Axler, 2015, Ch. 6). Two vectors $\vu, \vv \in \vecspace$ are called orthogonal if their inner product is zero, $\innerprod{\vu}{\vv} = 0$. The pair $(\vecspace, \innerprod{\cdot}{\cdot})$ is called an inner product space. Each inner product induces a norm via $\norm{\vu} \defeq \sqrt{\innerprod{\vu}{\vu}}$ for all $\vu \in \vecspace$, which in turn induces a metric via $\metric{\vu}{\vv} \defeq \norm{\vu - \vv}$ for all $\vu, \vv \in \vecspace$.

For the Euclidean space $\vecspace = \reals^{\featuredim}$, the standard inner product, also called the dot product, is $\innerprod{\featurevec}{\featurevec'} = \featurevec^{\top} \featurevec' = \sum_{\featureidx=1}^{\featuredim} \feature_{\featureidx} \feature'_{\featureidx}$ (Strang, 2016, Sect. 1.2). Geometrically, the inner product determines orthogonal projections: the orthogonal projection of $\featurevec'$ onto the direction of a nonzero vector $\featurevec$ has the signed length $\innerprod{\featurevec}{\featurevec'} / \norm{\featurevec}$ (Axler, 2015, Ch. 6) (see Fig. 1).

The inner product also defines the angle $\theta$ between two nonzero vectors $\featurevec, \featurevec' \in \reals^{\featuredim}$, via $\cos \theta \defeq \innerprod{\featurevec}{\featurevec'} / \big( \norm{\featurevec} \norm{\featurevec'} \big)$. The Cauchy-Schwarz inequality ensures that this ratio lies in $[-1, 1]$ (Axler, 2015, Ch. 6). For the Euclidean plane $\reals^{2}$, the angle defined in this way agrees with the elementary geometric angle between the two vectors (Strang, 2016, Sect. 1.2).

It is important to note that there are also other choices for an inner product on the Euclidean space $\reals^{\featuredim}$ (Axler, 2015, Ch. 6). As a case in point, each symmetric and positive definite matrix $\mA \in \reals^{\featuredim \times \featuredim}$ induces an inner product $\innerprod{\featurevec}{\featurevec'} \defeq \featurevec^{\top} \mA \featurevec'$. Different choices of an inner product result in different notions of similarity and, in turn, different notions of nearest neighbors (a numerical example is given in pythondemos/innerproduct.py).

Figure 1 of the entry innerproduct — Figure 1: Geometry of the standard inner product in the Euclidean space $\reals^{2}$: the orthogonal projection of $\featurevec'$ onto the direction of $\featurevec$ has the signed length $\innerprod{\featurevec}{\featurevec'} / \norm{\featurevec}$. The angle $\theta$ between the two vectors is defined via $\cos \theta \defeq \innerprod{\featurevec}{\featurevec'} / \big( \norm{\featurevec} \norm{\featurevec'} \big)$. The inner product measures the alignment of the two vectors and vanishes when they are orthogonal

Fig. 2 illustrates inner products between feature vectors in $\reals^{2}$ that represent five European cities. Each city is characterized by its latitude and longitude as its two features.

Figure 2 of the entry innerproduct — Figure 2: Five European cities, each represented by a feature vector $\featurevec = (\feature_{1}, \feature_{2})^{\top} \in \reals^{2}$ with latitude $\feature_{1}$ and longitude $\feature_{2}$ as features. Each city is drawn at the vector $\feature_{1} \vu + \feature_{2} \vv$ (filled markers) for a fixed orthonormal pair $\vu, \vv \in \reals^{2}$, chosen such that the vector of Helsinki (thick arrow) lies on the horizontal line and the other cities lie above it. The drawn arrows are therefore not the feature vectors $(\feature_{1}, \feature_{2})^{\top}$ themselves. However, since $\vu, \vv$ are orthonormal, the inner product of two vectors equals the standard inner product of the corresponding feature vectors, $\innerprod{\feature_{1} \vu + \feature_{2} \vv}{\feature_{1}' \vu + \feature_{2}' \vv} = \feature_{1} \feature_{1}' + \feature_{2} \feature_{2}'$. Each dashed ruler projects the vector of a city orthogonally onto the Helsinki vector. The open circles mark the resulting orthogonal projections. City coordinates retrieved from OpenStreetMap; positions computed by `pythondemos/innerproduct.py`

An inner product also gives rise to the notion of an orthonormal basis: a basis $\vb^{(1)}, \ldots, \vb^{(\featuredim)}$ of a vector space $\vecspace$ of dimension $\featuredim$ is orthonormal if its elements are pairwise orthogonal and have unit norm, $\innerprod{\vb^{(\featureidx)}}{\vb^{(\featureidx')}} = 0$ for $\featureidx \neq \featureidx'$ and $\innerprod{\vb^{(\featureidx)}}{\vb^{(\featureidx)}} = 1$ (Axler, 2015, Ch. 6). The coordinates of a vector with respect to an orthonormal basis are delivered by inner products: $\vu = \sum_{\featureidx=1}^{\featuredim} \expcoeff_{\featureidx} \vb^{(\featureidx)}$ with expansion coefficients $\expcoeff_{\featureidx} = \innerprod{\vu}{\vb^{(\featureidx)}}$. Conversely, an inner product can be defined by declaring a basis orthonormal: given any basis $\vb^{(1)}, \ldots, \vb^{(\featuredim)}$ of $\vecspace$, setting \[ \innerprod{\vu}{\vv} \defeq \sum_{\featureidx=1}^{\featuredim} \expcoeff_{\featureidx} \expcoeff'_{\featureidx} \quad \text{for } \vu = \sum_{\featureidx=1}^{\featuredim} \expcoeff_{\featureidx} \vb^{(\featureidx)} \text{ and } \vv = \sum_{\featureidx=1}^{\featuredim} \expcoeff'_{\featureidx} \vb^{(\featureidx)} \] yields the unique inner product for which this basis is orthonormal. The standard inner product of the Euclidean space arises in this way from the standard basis given by the columns of the identity matrix $\mathbf{I}_{\featuredim}$.

In machine learning (ML), an inner product serves as a measure of similarity. An embedding encodes the meaning of a text block as a vector in a vector space. The alignment of two text blocks is measured by an inner product of their corresponding embeddings (Goodfellow et al., 2016, Sect. 15.4). Each neuron of an artificial neural network (ANN) matches its input vector against a vector of learned model parameters (Goodfellow et al., 2016, Ch. 6). Attention scores in a large language model (LLM) are scaled inner products between query and key vectors (Vaswani et al., 2017).

These examples share a common pattern: many ML methods access the feature vectors of data points only through inner products. For a linear model, each hypothesis is of the form \[ \hypothesis(\featurevec) = \innerprod{\weights}{\featurevec} \text{,} \] which is the same template matching that a neuron applies to its input. The empirical risk minimization (ERM) objective function, the average loss on a training set $\trainset = \big\{ \big(\featurevec^{(\sampleidx)}, \truelabel^{(\sampleidx)}\big) \big\}_{\sampleidx=1}^{\samplesize}$, therefore depends on the feature vectors only through the inner products $\innerprod{\weights}{\featurevec^{(\sampleidx)}}$. A linear classifier such as the support vector machine (SVM) computes its prediction from the inner product $\innerprod{\weights}{\featurevec}$ between the model parameters and the feature vector, and kernel methods extend linear models to arbitrary feature spaces by evaluating inner products between transformed feature vectors via a kernel (Schölkopf and Smola, 2002). For all these ML methods, the choice of the inner product determines the notion of similarity in use and, in turn, the hypothesis learned from a training set.

Synonyms: scalar product, dot product.

See also: field, norm, vector, Hilbert space, orthogonality condition, metric space, Euclidean space, Cauchy-Schwarz inequality, kernel, neuron, attention, principal component analysis, $k$-nearest neighbors.

References

Axler (2015). Linear Algebra Done Right. Springer Nature.
Strang (2016). Introduction to Linear Algebra. Wellesley-Cambridge Press.
Goodfellow et al. (2016). Deep Learning. MIT Press.
Vaswani et al. (2017). Attention is all you need. Adv. Neural Inf. Process. Syst..
Schölkopf and Smola (2002). Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press.

Cite this entry

@misc{dictml_innerproduct,
  author = {Jung, Alexander},
  title = {inner product},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-08-06},
  url = {https://dictionaryofml.org/terms/innerproduct.html}
}