Dictionary of Applied Machine Learning: 114 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

feature learning

Updated on 2026-10-29

Readiness 100/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Markdown Cite this entry

See also feature hypothesis space principal component analysis autoencoder dimensionality reduction pretraining

Feature learning is the task of learning, from a dataset, a feature transformation that turns the raw features of a data point into new features that make a subsequent learning task easier. A feature learning method is specified by the two feature spaces, by a hypothesis space of candidate feature transformations, and by a quantitative measure of how useful a feature transformation is, computed from a dataset that need not carry labels. Principal component analysis (PCA) learns a linear feature transformation to fewer features, judged by the linear reconstruction error, and an autoencoder obtains a nonlinear one by the same criterion. The hidden layers of a deep net form a chain of feature transformations learned jointly with the output layer, so that each layer builds its features from those of the previous one. Pretraining an encoder on unlabeled data points learns features that are reused across learning tasks. In weather forecasting, forty measurements from the five days before are turned into two learned features from which linear regression predicts the maximum temperature of the next day.

Figure produced by the Python demo of the entry feature learning
Figure written by the entry's Python demo featlearn.py, which recomputes what the entry states: the script block by block · open in Colab

Definition

B-datasetConsider the task of forecasting the weather of tomorrow at a weather station from the measurements of the previous five days. Each day is a data point, and its label is the maximum temperature of that day. Its raw features are eight daily measurements: the lowest, highest and mean air temperature, the precipitation, the sunshine duration, the humidity, the air pressure and the wind speed. Taken over each of the five previous days, that is a list of forty numbers. A point with forty coordinates cannot be drawn in a scatterplot. A feature transformation that delivers two new features from the forty raw ones places every day in a scatterplot (see Fig. 1), in which the days whose label lies above the median of all labels separate from the others, and a linear model of the two new features may already predict the label. Feature learning is the task of learning such a feature transformation from a dataset instead of designing it by hand.

Figure 1 of the entry featlearn
Figure 1: Twenty of the 361 days of 2024 at Krems an der Donau, every 17th warm day (filled circles) and every 17th cold day (open squares) in calendar order, each placed by two learned features $z_{1}$ and $z_{2}$ in place of its forty raw ones, the eight daily measurements of each of the five previous days. Warm and cold days, split at the median of the label, separate along $z_{1}$, as they do for all 361 days. Data generated by pythondemos/featlearn.py
Feature learning is the task of learning a feature transformation \[{\mathbf \Phi}: \mathcal{X} \rightarrow \mathcal{X}', \qquad {\mathbf x} \mapsto {\mathbf z} := {\mathbf \Phi}({\mathbf x}) \text{.}\] The transformation maps the raw feature vector ${\mathbf x} \in \mathcal{X}$ of a data point to a new feature vector ${\mathbf z}$ from a new feature space $\mathcal{X}'$. A feature learning method is specified by three design choices: the two feature spaces $\mathcal{X}$ and $\mathcal{X}'$, a hypothesis space $\mathcal{H}$ of candidate feature transformations, and a quantitative measure of how useful a specific ${\mathbf \Phi} \in \mathcal{H}$ is. The measure is computed from a dataset, which need not carry labels, and it need not be the dataset of the learning task at hand. A useful feature transformation is one that makes a subsequent learning task easier (Goodfellow et al., 2016, Ch. 15): the data points become separable by a linear model, fewer features suffice, or the features learned once serve several learning tasks. In the weather example, the learned features are judged afterwards by the validation error of linear regression on them.

A measure of usefulness can be built from a second map. The feature transformation is then read as an encoder, and a decoder ${\mathbf \Phi}^{*}: \mathcal{X}' \rightarrow \mathcal{X}$ rebuilds the raw feature vector from the new one (Goodfellow et al., 2016, Ch. 14). What the encoder kept is then measured by how well the decoder rebuilds the data point from it, the reconstruction error \[ \frac{1}{m} \sum_{r=1}^{m} L\Big( {\mathbf x}^{(r)},\, {\mathbf \Phi}^{*}\big( {\mathbf \Phi}\big({\mathbf x}^{(r)}\big) \big) \Big) \text{.} \] The loss function $L$ penalizes a rebuilt feature vector for differing from the original. A common choice is the squared error loss (Goodfellow et al., 2016, Sect. 14.1). The dataset it is computed on needs no labels: what the decoder has to reproduce is the data point itself.

The decoder may be learned alongside the encoder. The measure is then the minimum of the reconstruction error over a second hypothesis space of decoders. It may instead be fixed, and the encoder alone must then supply what the decoder cannot. Restricting the decoder therefore decides what the new features have to carry.

B-pcaPrincipal component analysis (PCA) is this construction with $\mathcal{X} := \mathbb{R}^{d}$, $\mathcal{X}' := \mathbb{R}^{d'}$ for some $d' < d$, and the hypothesis space of linear maps, \[\mathcal{H} := \big\{ {\mathbf \Phi}: \mathbb{R}^{d} \rightarrow \mathbb{R}^{d'}: {\mathbf x} \mapsto {\mathbf W} {\mathbf x} \text{ with some } {\mathbf W} \in \mathbb{R}^{d' \times d} \big\} \text{.}\] The decoder is linear too and is learned alongside the encoder ${\mathbf W}$. The usefulness of a specific ${\mathbf \Phi}({\mathbf x}) = {\mathbf W} {\mathbf x}$ is therefore the minimum of the reconstruction error over the reconstruction matrices ${\mathbf R}$. On the dataset $\mathcal{D} = \big\{ {\mathbf x}^{(1)}, \ldots, {\mathbf x}^{(m)} \big\}$, that minimum is \[\min_{{\mathbf R} \in \mathbb{R}^{d \times d'}} \sum_{r=1}^{m} \left\lVert {\mathbf x}^{(r)} - {\mathbf R} {\mathbf W} {\mathbf x}^{(r)} \right\rVert_{2}^{2} \text{.}\] The rows of the best ${\mathbf W}$ are the $d'$ eigenvectors of the sample covariance matrix with the largest eigenvalues (Hastie et al., 2009, Sect. 14.5.1). An autoencoder keeps the same criterion and replaces both linear maps by artificial neural networks (ANNs), so that the new features come from a nonlinear feature transformation. With a linear decoder and the squared error loss, it spans the subspace PCA finds (Goodfellow et al., 2016, Sect. 14.1).

B-linregApplied to the weather example, PCA delivers the two learned features of Fig. 1. The forty raw features of the 361 days of 2024 at Krems an der Donau are turned into two learned features. Each raw feature is first scaled to zero sample mean and unit sample variance, because temperatures, precipitation and pressure carry different units. Warm and cold days separate along the first learned feature: 89 percent of the warm days have a positive $z_{1}$ and 92 percent of the cold days a nonpositive one. Linear regression on the two learned features is trained on the days from January to August. It then predicts the maximum temperature of the days from September to December with a validation error of 13.5 squared degrees Celsius. Linear regression on all forty raw features reaches 10.2, and predicting the sample mean of the training labels reaches 110.9. The two learned features therefore retain most of the predictive content of the forty raw ones.

A deep net learns a chain of feature transformations. Each hidden layer $\ell = 1, \ldots, L$ is a feature transformation ${\mathbf \Phi}^{(\ell)}$ that reads in the activations ${\mathbf z}^{(\ell-1)}$ of the previous layer, with ${\mathbf z}^{(0)} := {\mathbf x}$. The features delivered to the output layer are the composition ${\mathbf z}^{(L)} = {\mathbf \Phi}^{(L)} \circ \cdots \circ {\mathbf \Phi}^{(1)} ({\mathbf x})$ (see Fig. 2). The feature transformations are learned jointly with the output layer by empirical risk minimization (ERM) with the loss of the final prediction, so the usefulness of the learned features is measured by how well the output layer, typically a linear model, predicts the label from ${\mathbf z}^{(L)}$ (Goodfellow et al., 2016, Ch. 15). The chain is useful because each layer builds its features from those of the previous one, so the feature transformations ${\mathbf \Phi}^{(\ell)} \circ \cdots \circ {\mathbf \Phi}^{(1)}$ grow deeper with $\ell$. In image classification, early layers deliver edges, later ones corners and object parts (Goodfellow et al., 2016, Fig. 1.2), and the features at the last hidden layer separate the classes by a linear surface (Bishop and Bishop, 2024, Sect. 6.3.3).

Figure 2 of the entry featlearn
Figure 2: A deep net as a chain of feature transformations: hidden layer $\ell$ turns the activations $\vz^{(\ell-1)}$ of the previous layer into $\vz^{(\ell)}$, with $\vz^{(0)}$ the raw feature vector, and the output layer applies a linear model to $\vz^{(L)}$. All layers are learned jointly with the linear model by ERM with the loss of the final prediction
Pretraining an encoder on a large dataset of unlabeled data points, with a loss constructed from the data points themselves (see self-supervised learning), learns features that are reused across learning tasks (Goodfellow et al., 2016, Sect. 15.1). A new learning task appends a task-specific hypothesis with few model parameters to the learned ${\mathbf \Phi}$ and adapts it by fine-tuning on its own dataset. The dataset used to learn ${\mathbf \Phi}$ can then be much larger than any labeled dataset, because unlabeled data points are cheap to collect (Bishop and Bishop, 2024, Sect. 6.3.3).

Alternative constructions of a feature transformation do not learn ${\mathbf \Phi}$ (Goodfellow et al., 2016, Ch. 6). A kernel method fixes ${\mathbf \Phi}$ through the choice of a kernel, with $\mathcal{X}'$ the reproducing kernel Hilbert space (RKHS) $\mathcal{H}_{k}$ and ${\mathbf z} = k\big({\mathbf x},\cdot\big)$, the kernel with its first argument fixed, never evaluated explicitly, and learns only a linear hypothesis on $\mathcal{X}'$. A hand-designed ${\mathbf \Phi}$ encodes domain knowledge instead of a dataset.

See also: feature transformation, encoder, decoder, feature, feature space, hypothesis space, principal component analysis, autoencoder, dimensionality reduction, deep net, layer, pretraining.

References

  1. Goodfellow et al. (2016). Deep Learning. MIT Press.
  2. Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media. doi.org/10.1007/978-0-387-84858-7
  3. Bishop and Bishop (2024). Deep Learning: Foundations and Concepts. Springer Nature. doi.org/10.1007/978-3-031-45468-4

Cite this entry

@misc{dictml_featlearn,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {feature learning},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-03},
  url = {https://dictionaryofml.org/terms/featlearn.html}
}