Dictionary of Applied Machine Learning

Hilbert space

Typeset PDF version — the authoritative form of this entry

A Hilbert space is an inner product space that is complete: every Cauchy sequence of its elements has a limit that again belongs to the space. Two examples are the Euclidean space of a fixed dimension and the space of integrable random variables (RVs) with finite variance on a common probability space, where RVs are identified if they are equal with probability one and the expectation $\expect\{ x x' \}$ serves as the inner product. In machine learning (ML), Hilbert spaces are used to represent datasets and hypothesis spaces. As a case in point, a linear regression method uses a Euclidean space as the feature space and the hypothesis space.

Definition

A Hilbert space is an inner product space $(\hilbertspace, \innerprod{\cdot}{\cdot})$ that is complete: every Cauchy sequence of its elements has a limit that again belongs to $\hilbertspace$ (see Fig. 1). The inner product induces a norm $\normgeneric{\vu}{\hilbertspace} \defeq \sqrt{\innerprod{\vu}{\vu}}$ and, in turn, a metric $\metric{\vu}{\vv} \defeq \normgeneric{\vu - \vv}{\hilbertspace}$ (see inner product). A sequence $\vu^{(1)}, \vu^{(2)}, \ldots$ of elements of $\hilbertspace$ is a Cauchy sequence if for every $\epsilon > 0$ there is an index $N$ such that $\normgeneric{\vu^{(\sampleidx)} - \vu^{(\sampleidx')}}{\hilbertspace} < \epsilon$ for all $\sampleidx, \sampleidx' \geq N$ (Rudin, 1976, Definition 3.8). Completeness means that every such sequence has a limit $\lim_{\sampleidx \rightarrow \infty} \vu^{(\sampleidx)}$ that is itself an element of $\hilbertspace$ (Bauschke and Combettes, 2017, Ch. 2).

Figure 1 of the entry hilbertspace
Figure 1: A Hilbert space $\hilbertspace$ consists of vectors for which an inner product is defined. The inner product $\innerprod{\cdot}{\cdot}$ induces a norm $\normgeneric{\vu}{\hilbertspace} \defeq \sqrt{\innerprod{\vu}{\vu}}$ and, in turn, a metric $\metric{\vu}{\vv} \defeq \normgeneric{\vu - \vv}{\hilbertspace}$. Every Cauchy sequence $\vw^{(1)}, \vw^{(2)}, \ldots$ of vectors (dots) converges to another vector $\vw = \lim_{\sampleidx \rightarrow \infty} \vw^{(\sampleidx)}$ in the Hilbert space (cross). The dashed segment indicates the orthogonal projection of $\vv$ onto the subspace spanned by $\vu$: for unit-norm $\vu$, it intersects this subspace at the point $\innerprod{\vu}{\vv} \, \vu$. Data generated by pythondemos/hilbertspace.py
Two examples of Hilbert spaces are used throughout machine learning (ML). First, the Euclidean space $\reals^{\featuredim}$, for any fixed dimension $\featuredim$, is a Hilbert space with the standard inner product $\innerprod{\vu}{\vv} = \vu^{\top} \vv$. The completeness of $\reals^{\featuredim}$ follows from the completeness of the real numbers (Rudin, 1976, Theorem 3.11). Second, the set of all integrable random variables (RVs) $x$ with finite variance, defined on a common probability space, is a Hilbert space (Brockwell and Davis, 1991, Ch. 2). Here, two RVs are identified whenever the expectation of their squared difference is zero, $\expect\big\{ (x - x')^{2} \big\} = 0$, i.e., whenever $x = x'$ with probability one. With this identification, the expectation $\innerprod{x}{x'} \defeq \expect\{ x x' \}$ is an inner product, and the resulting space is complete (Brockwell and Davis, 1991, Ch. 2).

Hilbert spaces are the natural setting for ML methods that access feature vectors only through inner products. Examples of such methods are those using a linear model and, more generally, kernel methods. The hypothesis space of a kernel method can be represented by a reproducing kernel Hilbert space (RKHS): a Hilbert space of functions whose inner product reproduces point evaluations (Schölkopf and Smola, 2002; Hastie et al., 2009, Sect. 12.3).

The Hilbert space of integrable finite-variance RVs on a common probability space underlies optimal linear estimation. A linear estimator $\predictedlabel$ of an RV $\truelabel$ from an observed RV $\feature$ is an element of the subspace spanned by $\feature$. The corresponding estimation error $\truelabel - \predictedlabel$ is measured by the induced norm $\sqrt{\innerprod{\truelabel - \predictedlabel}{\truelabel - \predictedlabel}}$. The smallest error is obtained by the linear estimator whose error is orthogonal to $\feature$ (Brockwell and Davis, 1991, Ch. 2), \begin{align} 0 & = \innerprod{\truelabel - \predictedlabel}{\feature} \nonumber \\ & = \expect \big\{ \big(\truelabel - \predictedlabel \big) \feature \big\} \text{.} \nonumber \end{align} Thus, the error $\truelabel - \predictedlabel$ incurred by the optimal linear estimator $\predictedlabel$ is orthogonal to the observation $\feature$. When the RVs $\truelabel,\feature$ are zero-mean, the error $\truelabel - \predictedlabel$ becomes uncorrelated from $\feature$.

See also: inner product, vector space, norm, Cauchy sequence, Euclidean space, random variable, variance, expectation, reproducing kernel Hilbert space, kernel method, orthogonal projection.

References

  1. Rudin (1976). Principles of Mathematical Analysis. mcgraw.
  2. Bauschke and Combettes (2017). Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer Science+Business Media.
  3. Brockwell and Davis (1991). Time Series: Theory and Methods. Springer-Verlag.
  4. Schölkopf and Smola (2002). Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press.
  5. Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media.

Cite this entry

@misc{dictml_hilbertspace,
  author = {Jung, Alexander},
  title = {Hilbert space},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-08-06},
  url = {https://dictionaryofml.org/terms/hilbertspace.html}
}