# Hilbert space *Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-10-29. [HTML](https://dictionaryofml.org/terms/hilbertspace.html) · [typeset PDF](https://dictionaryofml.org/terms/hilbertspace.pdf) · [LaTeX](https://dictionaryofml.org/terms/hilbertspace.tex) · [Python demo](https://dictionaryofml.org/terms/hilbertspace.py) · [notebook](https://dictionaryofml.org/terms/hilbertspace.ipynb)* Math is LaTeX with the book's macros expanded, so no notation package is needed to read it. ## Abstract A Hilbert space is a complete inner product space: every Cauchy sequence of its elements has a limit in the space. Three examples in machine learning are: a finite-dimensional Euclidean space, a space of finite-variance RVs on a common probability space, and a reproducing kernel Hilbert space. Linear regression uses a Euclidean space as its feature space and to parameterize its hypothesis space. Optimal linear and nonlinear estimation of a random variable are orthogonal projections in the Hilbert space of finite-variance RVs. A kernel method uses a reproducing kernel Hilbert space as its transformed feature space and as its hypothesis space. ## Definition Consider gradient descent for linear regression, or any iterative training method whose hypothesis space is a metric space: it produces the sequence $h^{(1)}, h^{(2)}, \ldots$ of hypotheses. For the sequence to converge to an optimal hypothesis $h^{\star}$, the hypotheses must form a Cauchy sequence: for every distance $\epsilon > 0$, however small, there is an index $N$ such that the distance between $h^{(r)}$ and $h^{(r')}$ is below $\epsilon$ for all $r, r' \geq N$ (Rudin, 1976, Definition 3.8). Being a Cauchy sequence is not sufficient on its own. If the underlying metric space is complete, then every Cauchy sequence in that space converges to a point in the space. An example of a complete metric space is a Euclidean space $\mathbb{R}^{d}$ of finite dimension $d$. A Hilbert space is a generalization of a Euclidean space to possibly infinite dimensions. In particular, a Hilbert space $(\mathcal{H}, \langle \cdot, \cdot \rangle)$ is an inner product space that is complete, i.e., in which every Cauchy sequence of its elements has a limit that again belongs to $\mathcal{H}$ (Bauschke and Combettes, 2011, Sect. 1.12) (see Fig. 1). ![Figure 1 of the entry hilbertspace](https://dictionaryofml.org/terms/fig/hilbertspace-fig1.svg) *Figure 1: Completeness. The elements ${\mathbf w}^{(1)}, {\mathbf w}^{(2)}, \ldots$ of a Cauchy sequence (dots) lie ever closer to each other, and their limit ${\mathbf w} = \lim_{t \rightarrow \infty} {\mathbf w}^{(t)}$ (cross) again belongs to the Hilbert space. Data generated by pythondemos/hilbertspace.py* The elements of $\mathcal{H}$ are called vectors, whether they are arrays of numbers, RVs, or functions: what makes them vectors is that they can be added and scaled (see vector space). The inner product induces a norm $\left\lVert {\mathbf u} \right\rVert_{\mathcal{H}} := \sqrt{\langle {\mathbf u}, {\mathbf u} \rangle}$ and, in turn, a metric $d({\mathbf u},{\mathbf v}) := \left\lVert {\mathbf u} - {\mathbf v} \right\rVert_{\mathcal{H}}$ (see inner product), so a Cauchy sequence of $\mathcal{H}$ is one whose vectors eventually lie within any prescribed norm distance of each other. Three examples of Hilbert spaces are used throughout machine learning: a Euclidean space $\mathbb{R}^{d}$ as feature space, a space of RVs with finite variance on a common probability space, and a reproducing kernel Hilbert space of functions. The first of them carries the standard inner product $\langle {\mathbf u}, {\mathbf v} \rangle = {\mathbf u}^{\top} {\mathbf v}$, and its completeness follows from that of the real numbers (Rudin, 1976, Thm. 3.11). Linear regression uses the Euclidean space $\mathbb{R}^{d}$ in two roles. The feature vectors ${\mathbf x} \in \mathbb{R}^{d}$ of the data points it is applied to are vectors of this space, and so are the weights ${\mathbf w} \in \mathbb{R}^{d}$ that parameterize its hypothesis space $\big\{ h({\mathbf x}) = \langle {\mathbf w}, {\mathbf x} \rangle \, : \, {\mathbf w} \in \mathbb{R}^{d} \big\}$. Gradient descent searching for those weights produces a sequence ${\mathbf w}^{(1)}, {\mathbf w}^{(2)}, \ldots$ of model parameters, the sequence depicted in Fig. 1. These are vectors in the second role, so the condition of the opening paragraph is met here: $\mathbb{R}^{d}$ is complete, and a Cauchy sequence of model parameters converges to model parameters. The second example is the set of all RVs $x$ with finite variance, defined on a common probability space (Gray and Davisson, 2004, Sect. 5.8.1). Here, two RVs are identified whenever the expectation of their squared difference is zero, $\mathbb{E}\big\{ (x - x')^{2} \big\} = 0$, i.e., whenever $x = x'$ with probability one. With this identification, the expectation $\langle x, x' \rangle := \mathbb{E}\{ x x' \}$ is an inner product (Gray and Davisson, 2004, Sect. 5.8.1), and the resulting space is complete (Gray and Davisson, 2004, Lem. 5.1). In the Hilbert space of finite-variance RVs on a common probability space, optimal linear estimation is an orthogonal projection (see Fig. 2). A linear estimator $\widehat{y}$ of a random variable $y$ from an observed random variable $x$ is an element of the subspace $\big\{ a x \, : \, a \in \mathbb{R} \big\}$ spanned by $x$. The corresponding estimation error $y - \widehat{y}$ is measured by the induced norm $\sqrt{\langle y - \widehat{y}, y - \widehat{y} \rangle}$. The smallest error is obtained by the linear estimator whose error is orthogonal to $x$ (Gray and Davisson, 2004, Thm. 4.9), \begin{align} 0 & = \langle y - \widehat{y}, x \rangle \nonumber \\ & = \mathbb{E} \big\{ \big(y - \widehat{y} \big) x \big\} \text{.} \nonumber \end{align} When the RVs $y,x$ are zero-mean, this orthogonality means that the error $y - \widehat{y}$ is uncorrelated with $x$. ![Figure 2 of the entry hilbertspace](https://dictionaryofml.org/terms/fig/hilbertspace-fig2.svg) *Figure 2: Optimal linear estimation as an orthogonal projection. The random variable $y$, the observed random variable $x$, and the estimator $\widehat{y}$ are vectors of the Hilbert space of finite-variance RVs. The estimators that are linear in $x$ form the subspace $\{ a x : a \in \mathbb{R} \}$ spanned by $x$ (the straight line through it), and the estimation error $y - \widehat{y}$ is smallest for the $\widehat{y}$ whose error is orthogonal to $x$, the right angle marked by the small square at $\widehat{y}$* Optimal nonlinear estimation can also be represented as an orthogonal projection, but on a larger subspace. Projecting $y$ onto the RVs that are functions of $x$, rather than onto the multiples $a x$ alone, gives the conditional expectation $\mathbb{E}\{ y \mid x \}$: it has the smallest squared error among all estimators computed from $x$ (Gray and Davisson, 2004, Thm. 4.5). The third example is a reproducing kernel Hilbert space. It is a Hilbert space $\mathcal{H}$ of functions $h: \mathcal{X} \to \mathbb{R}$ whose inner product reproduces point evaluations. In particular, each reproducing kernel Hilbert space is associated with a kernel $k\big(\cdot,\cdot\big)$ such that $k\big({\mathbf x},\cdot\big) \in \mathcal{H}$ for every ${\mathbf x} \in \mathcal{X}$ and $$ h({\mathbf x}) = \langle h, k\big({\mathbf x},\cdot\big) \rangle \quad \text{for every } h \in \mathcal{H} \text{.} $$ Thus, each vector $h \in \mathcal{H}$ is itself a hypothesis map $\mathcal{X} \to \mathbb{R}$, evaluated by taking an inner product with $k\big({\mathbf x},\cdot\big)$. This construction of a hypothesis map generalizes the linear model, which uses the standard inner product of the Euclidean space. The map ${\mathbf x} \mapsto k\big({\mathbf x},\cdot\big)$ is a feature transformation, carrying the feature vector of a data point into $\mathcal{H}$. A kernel method uses a reproducing kernel Hilbert space in two roles. It is the transformed feature space, in which a data point is represented by $k\big({\mathbf x},\cdot\big)$, and it is the hypothesis space from which a hypothesis is learned (Hastie et al., 2009, Sect. 12.3.3; Schölkopf and Smola, 2002). These are the two roles of $\mathbb{R}^{d}$ in linear regression, with one difference: there a vector parameterizes a hypothesis, here a vector is one. The three examples differ in what their vectors are, not in their geometry. Any two Hilbert spaces whose orthonormal bases have the same number of elements are copies of one another: mapping one basis onto the other extends to a linear map that preserves inner products and, in turn, distances. Such a linear map is therefore an isometry (Rudin, 1976, Thms. 11.43 and 11.45). **See also:** [inner product](https://dictionaryofml.org/terms/innerproduct.html), vector space, [norm](https://dictionaryofml.org/terms/norm.html), Cauchy sequence, Euclidean space, reproducing kernel Hilbert space, orthogonal projection, isometry. ## References - Rudin (1976). Principles of Mathematical Analysis. *McGraw-Hill*. - Bauschke and Combettes (2011). Convex Analysis and Monotone Operator Theory in Hilbert Spaces. *Springer Science+Business Media*. - Gray and Davisson (2004). An Introduction to Statistical Signal Processing. *Cambridge Univ. Press*. - Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. - Schölkopf and Smola (2002). Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. *MIT Press*. ## Cite this entry ```bibtex @incollection{dictml_hilbertspace, author = {Jung, Alexander}, editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard}, title = {Hilbert space}, booktitle = {Dictionary of Applied Machine Learning}, publisher = {Springer Nature Singapore}, year = {2026}, doi = {10.1007/978-981-92-2012-0_26-1}, isbn = {978-981-92-2012-0}, url = {https://doi.org/10.1007/978-981-92-2012-0_26-1} } ```