Dictionary of Applied Machine Learning: 147 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

feature space

Updated on 2026-11-19

Readiness 88/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Markdown Cite this entry

See also feature vector feature feature transformation data point Hilbert space inner product hypothesis space feature learning kernel method kernel ridge regression

The feature space of a machine learning (ML) application is the set of all values that the feature vector of a data point can take. It is the range of the feature transformation that maps a data point to its feature vector. For data points with $d$ numeric features the feature space is usually the Euclidean space $\mathbb{R}^{d}$, whose Euclidean distance and inner product are what nearest neighbor methods and linear models use. The same data points placed in two feature spaces have different nearest neighbors and yield different hypotheses, so the choice of the feature space is part of the model, and dropping a feature is model pruning. A feature space can also be a finite set, such as the months of a year, or a set of strings or graphs, which a kernel makes accessible. A kernel brings a second feature space with it, a Hilbert space of possibly infinite dimension in which the kernel is the inner product and which a kernel method uses without ever forming its elements.

Figure produced by the Python demo of the entry feature space
Figure written by the entry's Python demo featurespace.py, which recomputes what the entry states: the script block by block · open in Colab

Definition

B-spacesA day at Krems an der Donau can be described by a single number, its morning minimum temperature, or by two numbers, that minimum and the maximum temperature of the previous day (GeoSphere Austria, 2026). The first description turns every day into a point of the real line $\mathbb{R}$, the second into a point of the plane $\mathbb{R}^{2}$. The set of all values that the feature vector of a data point can take is the feature space $\mathcal{X}$ of a machine learning (ML) application. A feature transformation \begin{equation} \label{equ_featurespace_map_dict} {\mathbf \Phi}: \mathcal{Z} \rightarrow \mathcal{X}, \quad {\mathbf x} = {\mathbf \Phi}({\mathbf z}) \text{,} \end{equation} maps a data point ${\mathbf z}$ from the set $\mathcal{Z}$ of all data points of the application to its feature vector, and the feature space is the range of this map (Shalev-Shwartz and Ben-David, 2014, Sect. 16.1). The two descriptions of a day are two feature transformations \(\eqref{equ_featurespace_map_dict}\) of the same data points, with the feature spaces $\mathbb{R}$ and $\mathbb{R}^{2}$.

For data points with $d$ numeric features, the default feature space is the Euclidean space $\mathbb{R}^{d}$ (Axler, 2024, Sect. 1A). The word space, rather than set, says that $\mathcal{X}$ carries structure that an ML method uses: the Euclidean distance $\left\lVert {\mathbf x} - {\mathbf x}' \right\rVert_{2}$ makes $\mathbb{R}^{d}$ a metric space, and the inner product ${\mathbf x}^{\top} {\mathbf x}'$ makes it a Hilbert space. The $k$-nearest neighbors ($k$-NN) method uses the distance: the nearest neighbor of a data point with feature vector ${\mathbf x}$ in a training set with feature vectors ${\mathbf x}^{(1)}, \ldots, {\mathbf x}^{(m)}$ is ${\mathbf x}^{(r^{\star})}$ with \begin{equation} \label{equ_featurespace_nn_dict} r^{\star} := \mathop{\mathrm{arg\,min}}_{r = 1, \ldots, m} \left\lVert {\mathbf x} - {\mathbf x}^{(r)} \right\rVert_{2} \text{,} \end{equation} and the label of that neighbor is the prediction for ${\mathbf x}$ (Cover and Hart, 1967; Hastie et al., 2009, Sect. 2.3.2). A linear model uses the inner product: its hypotheses are the maps $h({\mathbf x}) = {\mathbf w}^{\top} {\mathbf x} + b$ with model parameters ${\mathbf w} \in \mathbb{R}^{d}$ and $b \in \mathbb{R}$ (Hastie et al., 2009, Sect. 2.3.1).

B-linearWhich feature space the $366$ days of $2024$ at Krems are placed in changes what both methods deliver. The label of a day is its maximum daytime temperature. On $22$ May $2024$ the morning minimum was $14.4$ degrees Celsius, the maximum of the previous day $22.7$, and the maximum of the day itself $22.4$. In the feature space $\mathbb{R}$, the nearest neighbor \(\eqref{equ_featurespace_nn_dict}\) of this day among the other $365$ days is $4$ September, with the same morning minimum $14.4$ but a maximum of $34.1$. In $\mathbb{R}^{2}$ it is $11$ June, with the feature vector $(14.5, 22.7)^{\top}$ and a maximum of $20.2$ (Fig. 1). The prediction by the nearest neighbor misses the label by $11.7$ degrees in the first feature space and by $2.2$ degrees in the second, and $346$ of the $366$ days have a different nearest neighbor in $\mathbb{R}$ than in $\mathbb{R}^{2}$. A linear model learned by least squares from all $366$ days has a training mean squared error (MSE) of $19.2$ in $\mathbb{R}$ and of $9.2$ in $\mathbb{R}^{2}$.

B-finiteThe choice of the feature space is therefore part of the model. A hypothesis space is a set of maps $h: \mathcal{X} \rightarrow \mathcal{Y}$ into the label space $\mathcal{Y}$, so changing $\mathcal{X}$ changes which maps it contains. The hypotheses $h(x) = w x + b$ on $\mathbb{R}$ are, read as maps of the pair $(x_{1}, x_{2})^{\top}$, the hypotheses $w_{1} x_{1} + w_{2} x_{2} + b$ on $\mathbb{R}^{2}$ with $w_{2} = 0$. Dropping a feature thus shrinks the hypothesis space, which is model pruning, one of the three routes that regularization distinguishes: the training error of empirical risk minimization (ERM) cannot fall ($19.2$ against $9.2$), while the smaller hypothesis space may generalize better. Feature learning methods and feature standardization change the feature space in the other direction, by learning or rescaling the feature transformation \(\eqref{equ_featurespace_map_dict}\). Rescaling one feature changes the distances in the nearest neighbor rule \(\eqref{equ_featurespace_nn_dict}\), and with them the nearest neighbors.

Figure 1 of the entry featurespace
Figure 1: The same days of $2024$ at Krems in two feature spaces: every $18$th of the $366$ days (circles), $22$ May (filled square), its nearest neighbor in $\reals$, $4$ September (triangle), and its nearest neighbor in $\reals^{2}$, $11$ June (diamond), $23$ days in all. (a) The feature space $\reals$ of the morning minimum temperature, drawn against the label; the distance between two days is the horizontal gap between their markers, and the solid line is the linear model hypothesis learned by least squares. (b) The feature space $\reals^{2}$ of the morning minimum and the previous day's maximum; the filled square and the diamond almost coincide, while the triangle is far from both. Data generated by pythondemos/featurespace.py
A feature space need not be a Euclidean space. Describing a day by its month gives the finite feature space $\mathcal{X} = \{1, \ldots, 12\}$. A hypothesis on it is a table of $12$ values, and the table that minimizes the training MSE holds the monthly means of the maximum temperature, with an MSE of $17.0$. One-hot encoding (see feature vector) maps the $12$ months to the $12$ unit vectors of $\mathbb{R}^{12}$, and on these every table is a linear model: least squares learns the same $12$ means. Under the discrete metric, which is $0$ for two equal and $1$ for two different months, each of the $30$ other days of May is a nearest neighbor of $22$ May. Other feature spaces without arithmetic are the strings over an alphabet, for text documents and DNA sequences, and the graphs, for molecules. A method then accesses the feature space through a kernel that measures the similarity of two of its elements (Schölkopf and Smola, 2002, Sect. 13.2; Vishwanathan et al., 2010). The hours of a day, placed on a cycle graph, are the feature space of kernel ridge regression.

B-kernelA kernel brings a second feature space with it. A kernel $k: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ is an inner product of transformed feature vectors $\phi({\mathbf x})$ in a Hilbert space $\mathcal{H}$, $k\big({\mathbf x},{\mathbf x}'\big) = \langle \phi({\mathbf x}), \phi({\mathbf x}') \rangle_{\mathcal{H}}$ (Bishop, 2006, Eq. (6.1)), and $\mathcal{H}$ is also called the feature space of the kernel. For the polynomial kernel of degree two on $\mathbb{R}^{2}$, this space is explicit, \begin{equation} \label{equ_featurespace_poly_dict} k\big({\mathbf x},{\mathbf x}'\big) = \big(1 + {\mathbf x}^{\top} {\mathbf x}'\big)^{2} = \phi({\mathbf x})^{\top} \phi({\mathbf x}') \text{,} \end{equation} with the feature transformation $\phi({\mathbf x}) = \big(1, \sqrt{2} x_{1}, \sqrt{2} x_{2}, x_{1}^{2}, \sqrt{2} x_{1} x_{2}, x_{2}^{2}\big)^{\top} \in \mathbb{R}^{6}$, so a linear model on the transformed feature vectors is a quadratic hypothesis on the original feature space (Shalev-Shwartz and Ben-David, 2014, Example 16.1). The $366 \times 366$ matrix of the polynomial kernel values \(\eqref{equ_featurespace_poly_dict}\) of the days at Krems has rank $6$, the dimension of $\mathbb{R}^{6}$. The feature space of the Gaussian kernel $\exp\big(-\left\lVert {\mathbf x} - {\mathbf x}' \right\rVert_{2}^{2} / (2 \sigma^{2})\big)$ has infinite dimension (Shalev-Shwartz and Ben-David, 2014, Example 16.2): for $\sigma = 1$ degree, the matrix of its values on the $366$ days has rank $366$, so no feature space of smaller dimension produces these values. A kernel method never forms the transformed feature vectors; it uses the feature space $\mathcal{H}$ only through kernel values, which is what makes a space of infinite dimension usable (Bishop, 2006, Sect. 6.2; Shalev-Shwartz and Ben-David, 2014, Sect. 16.2). Fig. 2 draws three further shapes of a feature space: the real line, a bounded convex subset of the plane, and the nodes of an undirected graph. The second is the feature space of a sensor with a bounded range, and in the third the distance between two feature vectors is the length of the shortest path between their nodes.

Figure 2 of the entry featurespace
Figure 2: Three feature spaces. (a) The real line $\featurespace^{(1)} = \reals$. (b) A bounded convex set $\featurespace^{(2)} \subseteq \reals^{2}$. (c) A finite feature space $\featurespace^{(3)}$ whose elements are the nodes of an undirected graph

Synonyms: input space, instance space.

See also: feature vector, feature, feature transformation, data point, label space, Euclidean space, Euclidean distance, metric space, Hilbert space, inner product, $k$-nearest neighbors, linear model, hypothesis space, model pruning, feature learning, feature standardization, kernel, kernel method, kernel ridge regression, reproducing kernel Hilbert space, graph.

References

  1. GeoSphere Austria (2026). GeoSphere Austria climate station archive. dataset.api.hub.geosphere.at/v1/station/historical/klima-v2-1d
  2. Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. Cambridge Univ. Press. doi.org/10.1017/cbo9781107298019
  3. Axler (2024). Linear Algebra Done Right. Springer. doi.org/10.1007/978-3-031-41026-0
  4. Cover and Hart (1967). Nearest neighbor pattern classification. IEEE Transactions on Information Theory. doi.org/10.1109/TIT.1967.1053964
  5. Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media. doi.org/10.1007/978-0-387-84858-7
  6. Schölkopf and Smola (2002). Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press. doi.org/10.7551/mitpress/4175.001.0001
  7. Vishwanathan et al. (2010). Graph Kernels. Journal of Machine Learning Research.
  8. Bishop (2006). Pattern Recognition and Machine Learning. Springer Science+Business Media. doi.org/10.1007/978-0-387-45528-0

Cite this entry

@misc{dictml_featurespace,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {feature space},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-20},
  url = {https://dictionaryofml.org/terms/featurespace.html}
}