Dictionary of Applied Machine Learning

Hilbert space

Updated on 2026-09-17

Typeset PDF version — the authoritative form of this entry

Python demo — a script that recomputes what this entry states and prints one line per check

A Hilbert space is a complete inner product space: every Cauchy sequence of its elements has a limit in the space. Three examples in machine learning (ML) are: a finite-dimensional Euclidean space, a space of finite-variance random variables (RVs) on a common probability space, and a reproducing kernel Hilbert space (RKHS). Linear regression uses a Euclidean space as its feature space and to parameterize its hypothesis space. Optimal estimation amounts to a projection in a Hilbert space consisting of RVs. A kernel method uses an RKHS as its transformed feature space and as its hypothesis space.

Definition

B-cauchyConsider a machine learning (ML) method whose hypothesis space is a metric space. An iterative training method produces the sequence $\hypothesis^{(1)}, \hypothesis^{(2)}, \ldots$ of hypotheses. For the sequence to converge to an optimal hypothesis $\hypothesis^{\star}$, the hypotheses must form a Cauchy sequence. Being a Cauchy sequence is not sufficient on its own. If the underlying metric space is complete, then every Cauchy sequence in that space converges to a point in the space. A prime example of a complete metric space is a Euclidean space $\reals^{\nrfeatures}$ of finite dimension $\nrfeatures$.

B-rvspaceA Hilbert space is a generalization of a Euclidean space to possibly infinite dimensions. In particular, a Hilbert space $(\hilbertspace, \innerprod{\cdot}{\cdot})$ is an inner product space that is complete, i.e., in which every Cauchy sequence of its elements has a limit that again belongs to $\hilbertspace$ (Bauschke and Combettes, 2011, Sect. 1.12) (see Fig. 1).

Figure 1 of the entry hilbertspace
Figure 1: Completeness. The elements $\vw^{(1)}, \vw^{(2)}, \ldots$ of a Cauchy sequence (dots) lie ever closer to each other, and their limit $\vw = \lim_{\iteridx \rightarrow \infty} \vw^{(\iteridx)}$ (cross) again belongs to the Hilbert space. Data generated by pythondemos/hilbertspace.py
The elements of $\hilbertspace$ are called vectors, whether they are arrays of numbers, random variables (RVs), or functions: what makes them vectors is that they can be added and scaled (see vector space). The inner product induces a norm $\normgeneric{\vu}{\hilbertspace} \defeq \sqrt{\innerprod{\vu}{\vu}}$ and, in turn, a metric $\metric{\vu}{\vv} \defeq \normgeneric{\vu - \vv}{\hilbertspace}$ (see inner product). A sequence $\vu^{(1)}, \vu^{(2)}, \ldots$ of vectors of $\hilbertspace$ is a Cauchy sequence if its vectors eventually lie within any prescribed distance of each other: for every distance $\epsilon > 0$, however small, there is an index $N$ such that $\normgeneric{\vu^{(\sampleidx)} - \vu^{(\sampleidx')}}{\hilbertspace} < \epsilon$ for all $\sampleidx, \sampleidx' \geq N$ (Rudin, 1976, Definition 3.8).

Three examples of Hilbert spaces are used throughout ML: a Euclidean space $\reals^{\featuredim}$ as feature space, a space of RVs with finite variance on a common probability space, and a reproducing kernel Hilbert space (RKHS) of functions. The first of them carries the standard inner product $\innerprod{\vu}{\vv} = \vu^{\top} \vv$, and its completeness follows from that of the real numbers (Rudin, 1976, Thm. 3.11). Linear regression uses the Euclidean space $\reals^{\featuredim}$ in two roles. The feature vectors $\featurevec \in \reals^{\featuredim}$ of the data points it is applied to are vectors of this space, and so are the weights $\weights \in \reals^{\featuredim}$ that parameterize its hypothesis space $\big\{ \hypothesis(\featurevec) = \innerprod{\weights}{\featurevec} \, : \, \weights \in \reals^{\featuredim} \big\}$. Gradient descent (GD) searching for those weights produces a sequence $\weights^{(1)}, \weights^{(2)}, \ldots$ of model parameters, the sequence depicted in Fig. 1. These are vectors in the second role, so the condition of the opening paragraph is met here: $\reals^{\featuredim}$ is complete, and a Cauchy sequence of model parameters converges to model parameters.

The second example is the set of all RVs $x$ with finite variance, defined on a common probability space (Gray and Davisson, 2004, Sect. 5.8.1). Here, two RVs are identified whenever the expectation of their squared difference is zero, $\expect\big\{ (x - x')^{2} \big\} = 0$, i.e., whenever $x = x'$ with probability one. With this identification, the expectation $\innerprod{x}{x'} \defeq \expect\{ x x' \}$ is an inner product, and the resulting space is complete (Gray and Davisson, 2004, Lem. 5.1).

B-projIn the Hilbert space of finite-variance RVs on a common probability space, optimal linear estimation is an orthogonal projection (see Fig. 2). A linear estimator $\predictedlabel$ of an RV $\truelabel$ from an observed RV $\feature$ is an element of the subspace $\big\{ a \feature \, : \, a \in \reals \big\}$ spanned by $\feature$. The corresponding estimation error $\truelabel - \predictedlabel$ is measured by the induced norm $\sqrt{\innerprod{\truelabel - \predictedlabel}{\truelabel - \predictedlabel}}$. The smallest error is obtained by the linear estimator whose error is orthogonal to $\feature$ (Gray and Davisson, 2004, Thm. 4.9), \begin{align} 0 & = \innerprod{\truelabel - \predictedlabel}{\feature} \nonumber \\ & = \expect \big\{ \big(\truelabel - \predictedlabel \big) \feature \big\} \text{.} \nonumber \end{align} When the RVs $\truelabel,\feature$ are zero-mean, this orthogonality means that the error $\truelabel - \predictedlabel$ is uncorrelated with $\feature$.

Figure 2 of the entry hilbertspace
Figure 2: Optimal linear estimation as an orthogonal projection. The RV $\truelabel$, the observed RV $\feature$, and the estimator $\predictedlabel$ are vectors of the Hilbert space of finite-variance RVs. The estimators that are linear in $\feature$ form the subspace $\{ a \feature : a \in \reals \}$ spanned by $\feature$ (the straight line through it), and the estimation error $\truelabel - \predictedlabel$ is smallest for the $\predictedlabel$ whose error is orthogonal to $\feature$, the right angle marked by the small square at $\predictedlabel$
Optimal nonlinear estimation can also be represented as an orthogonal projection, but on a larger subspace. Projecting $\truelabel$ onto the RVs that are functions of $\feature$, rather than onto the multiples $a \feature$ alone, gives the conditional expectation $\expect\{ \truelabel \mid \feature \}$: it has the smallest squared error among all estimators computed from $\feature$ (Gray and Davisson, 2004, Thm. 4.5).

B-rkhsThe third example is an RKHS. It is a Hilbert space $\hilbertspace$ of functions $\hypothesis: \featurespace \to \reals$ whose inner product reproduces point evaluations. In particular, each RKHS is associated with a kernel $\kernelmap{\cdot}{\cdot}$ such that $\kernelmap{\featurevec}{\cdot} \in \hilbertspace$ for every $\featurevec \in \featurespace$ and \[ \hypothesis(\featurevec) = \innerprod{\hypothesis}{\kernelmap{\featurevec}{\cdot}} \quad \text{for every } \hypothesis \in \hilbertspace \text{.} \] Thus, each vector $\hypothesis \in \hilbertspace$ is itself a hypothesis map $\featurespace \to \reals$, evaluated by taking an inner product with $\kernelmap{\featurevec}{\cdot}$. This construction of a hypothesis map generalizes the linear model, which uses the standard inner product of the Euclidean space. The map $\featurevec \mapsto \kernelmap{\featurevec}{\cdot}$ is a feature transformation, carrying the feature vector of a data point into $\hilbertspace$. A kernel method uses an RKHS in two roles. It is the transformed feature space, in which a data point is represented by $\kernelmap{\featurevec}{\cdot}$, and it is the hypothesis space from which a hypothesis is learned (Hastie et al., 2009, Sect. 12.3.3; Schölkopf and Smola, 2002). These are the two roles of $\reals^{\featuredim}$ in linear regression, with one difference: there a vector parameterizes a hypothesis, here a vector is one.

Any two Hilbert spaces whose orthonormal bases have the same number of elements are copies of one another: mapping one basis onto the other extends to a linear map that preserves inner products and, in turn, distances. Such a linear map is therefore an isometry (Rudin, 1976, Thms. 11.43 and 11.45).

See also: inner product, vector space, norm, Cauchy sequence, Euclidean space, random variable, variance, expectation, conditional expectation, reproducing kernel Hilbert space, kernel method, orthogonal projection, isometry.

References

  1. Bauschke and Combettes (2011). Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer Science+Business Media. doi.org/10.1007/978-1-4419-9467-7
  2. Rudin (1976). Principles of Mathematical Analysis. McGraw-Hill.
  3. Gray and Davisson (2004). An Introduction to Statistical Signal Processing. Cambridge Univ. Press. ee.stanford.edu/~gray/sp.pdf
  4. Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media. doi.org/10.1007/978-0-387-84858-7
  5. Schölkopf and Smola (2002). Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press. doi.org/10.7551/mitpress/4175.001.0001

Cite this entry

@misc{dictml_hilbertspace,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {Hilbert space},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-09-21},
  url = {https://dictionaryofml.org/terms/hilbertspace.html}
}