Dictionary of Applied Machine Learning: 114 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

inner product

Updated on 2026-11-04

Readiness 100/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Markdown Cite this entry

See also norm Hilbert space

The inner product of two vectors in a vector space is a number that measures their similarity. Many machine learning (ML) applications represent a data point by a numeric array of features, such as the color intensities of an image, the amplitudes of a sensor signal, or the embeddings of the tokens used by large language models (LLMs). Each such array is a vector in some vector space. Many ML methods rely on measuring the similarity between the vectors that represent data points. Each neuron of an artificial neural network (ANN) matches its input against a learned template, and the attention unit of an LLM compares the query vector of one token with the key vector of another. Both comparisons compute an inner product.

Figure produced by the Python demo of the entry inner product
Figure written by the entry's Python demo innerproduct.py, which recomputes what the entry states: the script block by block · open in Colab

Definition

P-arrayThe color intensities of the pixels of an image, the amplitudes of a sensor signal at regular intervals, and the embedding of a token in a large language model (LLM) are numeric arrays, and data points are often characterized by features that form such an array. Such an array can be represented by a vector, i.e., an element of a vector space. Many machine learning (ML) methods compare two data points by measuring how similar the vectors that represent them are. One such measure of similarity is an inner product.

P-axiomsConsider a vector space $\mathcal{V}$ over the field of real numbers $\mathbb{R}$. An inner product in $\mathcal{V}$ is a function \[ \langle \cdot, \cdot \rangle: \mathcal{V} \times \mathcal{V} \to \mathbb{R} \] that satisfies the following properties for all vectors ${\mathbf u}, {\mathbf v}, {\mathbf w} \in \mathcal{V}$ and all scalars $\beta \in \mathbb{R}$ (Axler, 2024, Def. 6.2):

Over the field of complex numbers, symmetry becomes conjugate symmetry, $\langle {\mathbf u}, {\mathbf v} \rangle = \overline{\langle {\mathbf v}, {\mathbf u} \rangle}$, and the inner product is conjugate-linear in the second argument (Axler, 2024, Thm. 6.6); only real vector spaces are considered from here on. Two vectors ${\mathbf u}, {\mathbf v} \in \mathcal{V}$ are called orthogonal if their inner product is zero, $\langle {\mathbf u}, {\mathbf v} \rangle = 0$.

The pair $(\mathcal{V}, \langle \cdot, \cdot \rangle)$ is called an inner product space. Each inner product induces a norm via $\Vert {{\mathbf u}} \Vert := \sqrt{\langle {\mathbf u}, {\mathbf u} \rangle}$ for all ${\mathbf u} \in \mathcal{V}$, which in turn induces a metric via $d({\mathbf u},{\mathbf v}) := \Vert {{\mathbf u} - {\mathbf v}} \Vert$ for all ${\mathbf u}, {\mathbf v} \in \mathcal{V}$.

P-cosFor the Euclidean space $\mathcal{V} = \mathbb{R}^{d}$, the standard inner product, also called the dot product, is $\langle {\mathbf x}, {\mathbf x}' \rangle = {\mathbf x}^{\top} {\mathbf x}' = \sum_{j=1}^{d} x_{j} x'_{j}$ (Strang, 2016, Sect. 1.2). Geometrically, the inner product determines orthogonal projections: the orthogonal projection of ${\mathbf x}'$ onto the direction of a nonzero vector ${\mathbf x}$ has the signed length $\langle {\mathbf x}, {\mathbf x}' \rangle / \Vert {{\mathbf x}} \Vert$ (Axler, 2024, Example 6.56) (see Fig. 1).

Figure 1 of the entry innerproduct
Figure 1: Geometry of the standard inner product in the Euclidean space $\reals^{2}$. The dashed ruler drops from $\featurevec'$ perpendicularly onto the direction of $\featurevec$ and meets that direction at the orthogonal projection of $\featurevec'$. This orthogonal projection lies at the signed length $\innerprod{\featurevec}{\featurevec'} / \norm{\featurevec}$ from the origin, marked by the double arrow. The angle $\theta$ between the two vectors satisfies $\cos \theta = \innerprod{\featurevec}{\featurevec'} / \big( \norm{\featurevec} \norm{\featurevec'} \big)$ (defined below). The inner product measures the alignment of the two vectors and vanishes when they are orthogonal
The inner product also defines the angle $\theta$ between two nonzero vectors ${\mathbf x}, {\mathbf x}' \in \mathcal{V}$, via $\cos \theta := \langle {\mathbf x}, {\mathbf x}' \rangle / \big( \Vert {{\mathbf x}} \Vert \Vert {{\mathbf x}'} \Vert \big)$. The Cauchy-Schwarz inequality $|\langle {\mathbf x}, {\mathbf x}' \rangle| \leq \Vert {{\mathbf x}} \Vert \Vert {{\mathbf x}'} \Vert$ ensures that this ratio lies in $[-1, 1]$ (Axler, 2024, Thm. 6.14). Here $\cos$ is the cosine function, which decreases strictly from $\cos 0 = 1$ to $\cos \pi = -1$ and therefore takes each value of $[-1,1]$ exactly once on the interval $[0, \pi]$. For the Euclidean plane $\mathbb{R}^{2}$, as illustrated by Fig. 1, the angle defined in this way agrees with the elementary geometric angle between two vectors (Strang, 2016, Sect. 1.2).

For two vectors with given norms, the sign and magnitude of their inner product are determined by the angle $\theta$ between them. The inner product is positive for $\theta < \pi/2$, zero for $\theta = \pi/2$, where the two vectors are orthogonal, and negative for $\theta > \pi/2$. The angle drawn in Fig. 1 is smaller than $\pi/2$: the two vectors have a positive inner product, and the orthogonal projection therefore falls on the same side of the origin as ${\mathbf x}$.

P-projectFig. 1 projects one vector onto the direction of another, that is, onto the line it spans. The same characterization holds for a whole subspace, and it is what turns approximation problems, with respect to the induced norm, into systems of linear equations. Consider a subspace $\mathcal{U} \subseteq \mathcal{V}$ and a vector ${\mathbf v} \in \mathcal{V}$. In general, the vector of $\mathcal{U}$ closest to ${\mathbf v}$ with respect to some norm is called the projection of ${\mathbf v}$ onto $\mathcal{U}$. If the norm is induced by some inner product, the projection becomes an orthogonal projection. A vector $\widehat{{\mathbf v}} \in \mathcal{U}$ minimizes $\Vert {{\mathbf v} - {\mathbf u}} \Vert$ over all ${\mathbf u} \in \mathcal{U}$ if and only if the approximation error ${\mathbf v} - \widehat{{\mathbf v}}$ is orthogonal to every vector in $\mathcal{U}$, \begin{equation} \label{equ_orthogonality_innerproduct_dict} \langle {\mathbf v} - \widehat{{\mathbf v}}, {\mathbf u} \rangle = 0 \quad \text{for all } {\mathbf u} \in \mathcal{U} \text{.} \end{equation} The reason is the Pythagorean identity \[ \Vert {{\mathbf v} - {\mathbf u}} \Vert^{2} = \Vert {{\mathbf v} - \widehat{{\mathbf v}}} \Vert^{2} + \Vert {\widehat{{\mathbf v}} - {\mathbf u}} \Vert^{2} \text{,} \] which holds for every ${\mathbf u} \in \mathcal{U}$ whenever \(\eqref{equ_orthogonality_innerproduct_dict}\) does. The second term is non-negative, so no vector of $\mathcal{U}$ is closer to ${\mathbf v}$ than $\widehat{{\mathbf v}}$ is: orthogonality of the error and being closest are the same condition (Axler, 2024, Thm. 6.61). The orthogonality condition \(\eqref{equ_orthogonality_innerproduct_dict}\) is linear in $\widehat{{\mathbf v}}$, so a minimizer can be computed by solving linear equations.

P-convexFig. 2 illustrates inner products between feature vectors in $\mathbb{R}^{2}$ that represent five European cities. Each city is characterized by its latitude and longitude as its two features, and drawn at the vector $x_{1} {\mathbf u} + x_{2} {\mathbf v}$ for a fixed orthonormal pair ${\mathbf u}, {\mathbf v} \in \mathbb{R}^{2}$. The orthogonal projections of the other cities onto the direction of Helsinki rank them by their alignment with it.

Figure 2 of the entry innerproduct
Figure 2: Five European cities, each represented by a feature vector $\featurevec = (\feature_{1}, \feature_{2})^{\top} \in \reals^{2}$ with latitude $\feature_{1}$ and longitude $\feature_{2}$ as features. Each city is drawn at the vector $\feature_{1} \vu + \feature_{2} \vv$ (filled circles) for a fixed orthonormal pair $\vu, \vv \in \reals^{2}$, chosen such that the vector of Helsinki (thick arrow) lies on the horizontal line and the other cities lie above it. The drawn arrows are therefore not the feature vectors $(\feature_{1}, \feature_{2})^{\top}$ themselves. However, since $\vu, \vv$ are orthonormal, the inner product of two vectors equals the standard inner product of the corresponding feature vectors, $\innerprod{\feature_{1} \vu + \feature_{2} \vv}{\feature_{1}' \vu + \feature_{2}' \vv} = \feature_{1} \feature_{1}' + \feature_{2} \feature_{2}'$. Each dashed ruler projects the vector of a city orthogonally onto the Helsinki vector. The open circles mark the resulting orthogonal projections. City coordinates retrieved from OpenStreetMap
The characterization extends beyond subspaces. For a non-empty closed convex set $\mathcal{C} \subseteq \mathbb{R}^{d}$, a vector $\widehat{{\mathbf v}} \in \mathcal{C}$ is the projection of ${\mathbf v}$ onto $\mathcal{C}$ if and only if \begin{equation} \label{equ_variational_inequality_dict} \langle {\mathbf v} - \widehat{{\mathbf v}}, {\mathbf u} - \widehat{{\mathbf v}} \rangle \leq 0 \quad \text{for all } {\mathbf u} \in \mathcal{C} \text{.} \end{equation} Such a projection exists and is unique (Bertsekas, 2009, Prop. 1.1.9).

P-linregOne application of the subspace case \(\eqref{equ_orthogonality_innerproduct_dict}\) is a characterization of linear regression. The learned model parameters are those for which the prediction error is orthogonal to every column of the feature matrix (see normal equations). The resulting predictions are therefore the orthogonal projection of the label vector onto the column space of the feature matrix. The convex case \(\eqref{equ_variational_inequality_dict}\) becomes relevant when linear regression is modified by adding a convex constraint on the model parameters.

P-weightThe Euclidean space $\mathbb{R}^{d}$ carries other inner products besides the standard one (Axler, 2024, Example 6.3). For example, each symmetric and positive definite (pd) matrix ${\mathbf A} \in \mathbb{R}^{d \times d}$, i.e., one with ${\mathbf x}^{\top} {\mathbf A} {\mathbf x} > 0$ for every nonzero ${\mathbf x}$, induces an inner product $\langle {\mathbf x}, {\mathbf x}' \rangle := {\mathbf x}^{\top} {\mathbf A} {\mathbf x}'$. Such a matrix weighs the coordinate directions against each other, and thereby fixes a different notion of similarity between data points. Take ${\mathbf A} = \operatorname{diag}(4, 1/4)$, which counts the first feature sixteen times as heavily as the second, and the query ${\mathbf x} = (1, 1)^{\top}$. Of the two candidates $(0.9, 1.2)^{\top}$ and $(1.3, 0.4)^{\top}$, the standard inner product is larger for the first ($2.1$ against $1.7$) and the weighted one larger for the second ($5.3$ against $3.9$): the same query and the same candidates, but a different answer to which is more similar. Since an inner product also induces a norm and a metric, the choice of ${\mathbf A}$ fixes which data points count as nearest neighbors as well.

P-basisAn inner product also gives rise to the notion of an orthonormal basis. Consider a basis ${\mathbf b}^{(1)}, \ldots, {\mathbf b}^{(d)}$ of a vector space $\mathcal{V}$ of dimension $d$. It is orthonormal if its elements are pairwise orthogonal and have unit norm, $\langle {\mathbf b}^{(j)}, {\mathbf b}^{(j')} \rangle = 0$ for $j \neq j'$ and $\langle {\mathbf b}^{(j)}, {\mathbf b}^{(j)} \rangle = 1$ (Axler, 2024, Def. 6.22). The coordinates of a vector with respect to an orthonormal basis are delivered by inner products: ${\mathbf u} = \sum_{j=1}^{d} \beta_{j} {\mathbf b}^{(j)}$ with expansion coefficients $\beta_{j} = \langle {\mathbf u}, {\mathbf b}^{(j)} \rangle$. Conversely, an inner product can be defined by declaring a basis orthonormal. Take any basis ${\mathbf b}^{(1)}, \ldots, {\mathbf b}^{(d)}$ of $\mathcal{V}$. Then setting \[ \langle {\mathbf u}, {\mathbf v} \rangle := \sum_{j=1}^{d} \beta_{j} \beta'_{j} \quad \text{for } {\mathbf u} = \sum_{j=1}^{d} \beta_{j} {\mathbf b}^{(j)} \text{ and } {\mathbf v} = \sum_{j=1}^{d} \beta'_{j} {\mathbf b}^{(j)} \] yields the unique inner product for which this basis is orthonormal. The standard inner product of the Euclidean space arises in this way from the standard basis given by the columns of the identity matrix $\mathbf{I}_{d}$.

Several ML constructions use the similarity measure of the opening paragraph. An embedding maps a text block to a vector in a vector space. The alignment of two text blocks is measured by an inner product of their corresponding embeddings (Goodfellow et al., 2016, Sect. 12.4.2). Each neuron of an artificial neural network (ANN) matches its input vector against a vector of learned model parameters (Goodfellow et al., 2016, Ch. 6). Attention scores in an LLM are scaled inner products between query and key vectors (Vaswani et al., 2017).

Many ML methods access the feature vectors of data points only through inner products between feature vectors. A linear model uses hypotheses $h({\mathbf x}) = \langle {\mathbf w}, {\mathbf x} \rangle$. Consider regularized empirical risk minimization (RERM) over a training set $\mathcal{D}^{(\rm train)} = \big\{ \big({\mathbf x}^{(r)}, y^{(r)}\big) \big\}_{r=1}^{m}$, with a penalty that increases strictly with $\Vert {{\mathbf w}} \Vert$. It returns model parameters that are a linear combination of the feature vectors in the training set, $\widehat{{\mathbf w}} = \sum_{r=1}^{m} \beta_{r} {\mathbf x}^{(r)}$. This is the representer theorem, and the support-vector expansion of the support vector machine (SVM) is a special case of it (Lampert, 2009, Thm. 2.3; Schölkopf and Smola, 2002).

P-kernelThe prediction for a new data point then reads \[ \widehat{h}({\mathbf x}) = \sum_{r=1}^{m} \beta_{r} \langle {\mathbf x}^{(r)}, {\mathbf x} \rangle \text{.} \] Training and prediction therefore use the feature vectors only through inner products. Within the training set these are $\langle {\mathbf x}^{(r)}, {\mathbf x}^{(r')} \rangle$, and between the training set and the new data point they are $\langle {\mathbf x}^{(r)}, {\mathbf x} \rangle$. This is used by kernel methods that use a kernel $k({\mathbf x}, {\mathbf x}')$ to compute inner products without ever forming feature vectors (Schölkopf and Smola, 2002) (see kernel method).

P-distMethods based on Euclidean distances access the feature vectors only through inner products as well, even when they are defined without reference to one. The squared Euclidean distance between two feature vectors is a sum of three inner products, $\left\lVert {\mathbf x} - {\mathbf x}' \right\rVert_{2}^{2} = \langle {\mathbf x}, {\mathbf x} \rangle - 2 \langle {\mathbf x}, {\mathbf x}' \rangle + \langle {\mathbf x}', {\mathbf x}' \rangle$. A method that uses the feature vectors only through their Euclidean distances therefore also uses them only through inner products. $k$-nearest neighbors ($k$-NN) is one example: it ranks the data points in the training set by the distance of their feature vectors to the feature vector of the new data point (Hastie et al., 2009, Sect. 2.3.2). The same kernel substitution therefore applies to it: by the identity above, each Euclidean distance is computed from three kernel evaluations, and no feature vector is ever formed.

Synonyms: scalar product, dot product.

See also: norm, vector, Hilbert space, orthogonality condition, Euclidean space, Cauchy-Schwarz inequality, orthogonal projection, kernel.

References

  1. Axler (2024). Linear Algebra Done Right. Springer. doi.org/10.1007/978-3-031-41026-0
  2. Strang (2016). Introduction to Linear Algebra. Wellesley-Cambridge Press.
  3. Bertsekas (2009). Convex Optimization Theory. Athena Scientific.
  4. Goodfellow et al. (2016). Deep Learning. MIT Press.
  5. Vaswani et al. (2017). Attention is all you need. Adv. Neural Inf. Process. Syst.. papers.nips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
  6. Lampert (2009). Kernel Methods in Computer Vision. Found. Trends Comput. Graph. Vis.. doi.org/10.1561/0600000027
  7. Schölkopf and Smola (2002). Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press. doi.org/10.7551/mitpress/4175.001.0001
  8. Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media. doi.org/10.1007/978-0-387-84858-7

Cite this entry

@misc{dictml_innerproduct,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {inner product},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-04},
  url = {https://dictionaryofml.org/terms/innerproduct.html}
}