Dictionary of Applied Machine Learning: 147 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here
Updated on 2026-11-19
Readiness 88/100
See also feature vector feature feature transformation data point Hilbert space inner product hypothesis space feature learning kernel method kernel ridge regression
The feature space of a machine learning (ML) application is the set of all values that the feature vector of a data point can take. It is the range of the feature transformation that maps a data point to its feature vector. For data points with $d$ numeric features the feature space is usually the Euclidean space $\mathbb{R}^{d}$, whose Euclidean distance and inner product are what nearest neighbor methods and linear models use. The same data points placed in two feature spaces have different nearest neighbors and yield different hypotheses, so the choice of the feature space is part of the model, and dropping a feature is model pruning. A feature space can also be a finite set, such as the months of a year, or a set of strings or graphs, which a kernel makes accessible. A kernel brings a second feature space with it, a Hilbert space of possibly infinite dimension in which the kernel is the inner product and which a kernel method uses without ever forming its elements.

featurespace.py, which recomputes what the entry states: the script block by block · open in ColabB-spacesA day at Krems an der Donau can be described by a single number, its morning minimum temperature, or by two numbers, that minimum and the maximum temperature of the previous day (GeoSphere Austria, 2026). The first description turns every day into a point of the real line $\mathbb{R}$, the second into a point of the plane $\mathbb{R}^{2}$. The set of all values that the feature vector of a data point can take is the feature space $\mathcal{X}$ of a machine learning (ML) application. A feature transformation \begin{equation} \label{equ_featurespace_map_dict} {\mathbf \Phi}: \mathcal{Z} \rightarrow \mathcal{X}, \quad {\mathbf x} = {\mathbf \Phi}({\mathbf z}) \text{,} \end{equation} maps a data point ${\mathbf z}$ from the set $\mathcal{Z}$ of all data points of the application to its feature vector, and the feature space is the range of this map (Shalev-Shwartz and Ben-David, 2014, Sect. 16.1). The two descriptions of a day are two feature transformations \(\eqref{equ_featurespace_map_dict}\) of the same data points, with the feature spaces $\mathbb{R}$ and $\mathbb{R}^{2}$.
For data points with $d$ numeric features, the default feature space is the Euclidean space $\mathbb{R}^{d}$ (Axler, 2024, Sect. 1A). The word space, rather than set, says that $\mathcal{X}$ carries structure that an ML method uses: the Euclidean distance $\left\lVert {\mathbf x} - {\mathbf x}' \right\rVert_{2}$ makes $\mathbb{R}^{d}$ a metric space, and the inner product ${\mathbf x}^{\top} {\mathbf x}'$ makes it a Hilbert space. The $k$-nearest neighbors ($k$-NN) method uses the distance: the nearest neighbor of a data point with feature vector ${\mathbf x}$ in a training set with feature vectors ${\mathbf x}^{(1)}, \ldots, {\mathbf x}^{(m)}$ is ${\mathbf x}^{(r^{\star})}$ with \begin{equation} \label{equ_featurespace_nn_dict} r^{\star} := \mathop{\mathrm{arg\,min}}_{r = 1, \ldots, m} \left\lVert {\mathbf x} - {\mathbf x}^{(r)} \right\rVert_{2} \text{,} \end{equation} and the label of that neighbor is the prediction for ${\mathbf x}$ (Cover and Hart, 1967; Hastie et al., 2009, Sect. 2.3.2). A linear model uses the inner product: its hypotheses are the maps $h({\mathbf x}) = {\mathbf w}^{\top} {\mathbf x} + b$ with model parameters ${\mathbf w} \in \mathbb{R}^{d}$ and $b \in \mathbb{R}$ (Hastie et al., 2009, Sect. 2.3.1).
B-linearWhich feature space the $366$ days of $2024$ at Krems are placed in changes what both methods deliver. The label of a day is its maximum daytime temperature. On $22$ May $2024$ the morning minimum was $14.4$ degrees Celsius, the maximum of the previous day $22.7$, and the maximum of the day itself $22.4$. In the feature space $\mathbb{R}$, the nearest neighbor \(\eqref{equ_featurespace_nn_dict}\) of this day among the other $365$ days is $4$ September, with the same morning minimum $14.4$ but a maximum of $34.1$. In $\mathbb{R}^{2}$ it is $11$ June, with the feature vector $(14.5, 22.7)^{\top}$ and a maximum of $20.2$ (Fig. 1). The prediction by the nearest neighbor misses the label by $11.7$ degrees in the first feature space and by $2.2$ degrees in the second, and $346$ of the $366$ days have a different nearest neighbor in $\mathbb{R}$ than in $\mathbb{R}^{2}$. A linear model learned by least squares from all $366$ days has a training mean squared error (MSE) of $19.2$ in $\mathbb{R}$ and of $9.2$ in $\mathbb{R}^{2}$.
B-finiteThe choice of the feature space is therefore part of the
model. A hypothesis space is a set of maps $h:
\mathcal{X} \rightarrow \mathcal{Y}$ into the label space
$\mathcal{Y}$, so changing $\mathcal{X}$ changes which maps it
contains. The hypotheses $h(x) = w x
+ b$ on $\mathbb{R}$ are, read as maps of the pair $(x_{1},
x_{2})^{\top}$, the hypotheses $w_{1} x_{1} +
w_{2} x_{2} + b$ on $\mathbb{R}^{2}$ with $w_{2} = 0$. Dropping a
feature thus shrinks the hypothesis space, which is
model pruning, one of the three routes that
regularization distinguishes: the training error of empirical risk minimization (ERM)
cannot fall ($19.2$ against $9.2$), while the smaller hypothesis space
may generalize better. Feature learning methods and
feature standardization change the feature space in the other
direction, by learning or rescaling the feature transformation
\(\eqref{equ_featurespace_map_dict}\). Rescaling one feature
changes the distances in the nearest neighbor rule
\(\eqref{equ_featurespace_nn_dict}\), and with them the nearest
neighbors.
pythondemos/featurespace.py
B-kernelA kernel brings a second feature space with it. A kernel
$k: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ is
an inner product of transformed feature vectors
$\phi({\mathbf x})$ in a Hilbert space $\mathcal{H}$,
$k\big({\mathbf x},{\mathbf x}'\big) =
\langle \phi({\mathbf x}), \phi({\mathbf x}') \rangle_{\mathcal{H}}$
(Bishop, 2006, Eq. (6.1)), and $\mathcal{H}$ is also called
the feature space of the kernel. For the polynomial
kernel of degree two on $\mathbb{R}^{2}$, this space is explicit,
\begin{equation}
\label{equ_featurespace_poly_dict}
k\big({\mathbf x},{\mathbf x}'\big)
= \big(1 + {\mathbf x}^{\top} {\mathbf x}'\big)^{2}
= \phi({\mathbf x})^{\top} \phi({\mathbf x}')
\text{,}
\end{equation}
with the feature transformation $\phi({\mathbf x}) =
\big(1, \sqrt{2} x_{1}, \sqrt{2} x_{2},
x_{1}^{2}, \sqrt{2} x_{1} x_{2},
x_{2}^{2}\big)^{\top} \in \mathbb{R}^{6}$, so a linear model
on the transformed feature vectors is a quadratic
hypothesis on the original feature space
(Shalev-Shwartz and Ben-David, 2014, Example 16.1). The $366 \times 366$
matrix of the polynomial kernel values
\(\eqref{equ_featurespace_poly_dict}\) of the days at Krems has
rank $6$, the dimension of $\mathbb{R}^{6}$. The feature
space of the Gaussian kernel $\exp\big(-\left\lVert {\mathbf x}
- {\mathbf x}' \right\rVert_{2}^{2} / (2 \sigma^{2})\big)$ has infinite
dimension (Shalev-Shwartz and Ben-David, 2014, Example 16.2): for $\sigma = 1$
degree, the matrix of its values on the $366$ days has
rank $366$, so no feature space of smaller dimension
produces these values. A kernel method never forms the
transformed feature vectors; it uses the feature space
$\mathcal{H}$ only through kernel values, which is what
makes a space of infinite dimension usable
(Bishop, 2006, Sect. 6.2; Shalev-Shwartz and Ben-David, 2014, Sect. 16.2).
Fig. 2 draws three further shapes of a
feature space: the real line, a bounded convex subset of the
plane, and the nodes of an undirected graph. The second is the
feature space of a sensor with a bounded range, and in the third the
distance between two feature vectors is the length of the
shortest path between their nodes.
Synonyms: input space, instance space.
See also: feature vector, feature, feature transformation, data point, label space, Euclidean space, Euclidean distance, metric space, Hilbert space, inner product, $k$-nearest neighbors, linear model, hypothesis space, model pruning, feature learning, feature standardization, kernel, kernel method, kernel ridge regression, reproducing kernel Hilbert space, graph.
@misc{dictml_featurespace,
author = {Jung, Alexander},
editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
title = {feature space},
howpublished = {Dictionary of Applied Machine Learning (course edition)},
year = {2026},
doi = {10.5281/zenodo.21569296},
note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-20},
url = {https://dictionaryofml.org/terms/featurespace.html}
}