Dimensionality reduction uses a map
$h: \mathbb{R}^{d} \to \mathbb{R}^{d'}$ to
replace a raw feature vector ${\mathbf x} = (x_{1}, \,\ldots,
\,x_{d})^{\top}$ by a shorter feature vector
${\mathbf z} = (z_{1}, \,\ldots, \,z_{d'})^{\top}$, with
$d' < d$. Using fewer features lowers
the effective dimension of a model, which reduces the chance of
overfitting. Moreover, it lowers the amount of computation
needed for modeltraining and for inference. It also enables visualization: two features
can be used as the coordinates in a scatterplot. Principal component analysis (PCA)
learns a linear map by minimizing the reconstruction error, while a
random projection uses a linear map drawn at random, without
any training. An autoencoder learns a nonlinear map
represented by an artificial neural network (ANN).
B-patchesA square patch of the
photograph in Fig. 2 is described by
$48$ numbers, and two numbers computed from them suffice to tell
sky from vegetation. Dimensionality reduction uses
a map $h$, a transformation $h: \mathbb{R}^{d} \rightarrow \mathbb{R}^{d'}$
to replace the raw feature vector ${\mathbf x} = (x_{1}, \,\ldots, \,x_{d})^{\top}$
of a data point
by a shorter feature vector
${\mathbf z} = (z_{1}, \,\ldots, \,z_{d'})^{\top}$, with
$d' < d$ (Fig. 1).
Using a shorter feature vector is beneficial in several ways:
Computational benefit: Using fewer features lowers the amount of
computation needed for modeltraining and for
inference. For example, linear regression methods
need to invert a matrix whose size is determined by the number of features.
Visualization: A map can be learned that delivers two
features $z_{1},z_{2}$, which can then be used as the coordinates in a
scatterplot. Each square patch of a photograph is first
averaged down to a $4 \times 4$ grid of colors, still too many
numbers to plot; Fig. 2 then places
each patch at the two features that principal component analysis (PCA) learns for
it, which keep most of the variance of the raw
feature vectors.
Figure 1: Dimensionality reduction learns a map $\hypothesis$ that
sends the raw feature vector $\featurevec$ of a
data point, a list of $\featuredim$ numbers, to a shorter
feature vector $\vz$ of $\featuredim' < \featuredim$ numbersFigure 2: Top: the photograph of the Oetscher massif with the patch
grid drawn on it; each $128 \times 128$ pixel patch is one
data point. Bottom: every patch placed at the two
features $z_{1},
z_{2}$ that PCA learns for it. The raw feature vector
of a patch is a $4 \times 4$ grid of colors, $48$ numbers; the
two learned features keep $86$ percent of its
variance. A patch is drawn as an open circle when its
blue channel exceeds the other two by a fixed margin (sky)
and as a filled triangle when its green channel does
(vegetation); the remaining patches, the rock face and the
mixed ones along the tree line, are small dots. Sky and
vegetation lie in separate regions, with $93$ percent of them
on their own side of the midpoint between the two centers.
Data generated by pythondemos/dimred.py
Dimensionality reduction methods differ in the hypothesis space used for
the map, in the criterion they learn it by, and in whether the map
is learned from data at all. PCA learns a linear map:
the map is a
matrix chosen so that the reconstruction error, the squared distance
between ${\mathbf x}$ and the feature vector reconstructed from ${\mathbf z}$ by
a second linear map, is minimal on average over the training set
(Hotelling, 1933). An autoencoder learns a nonlinear map
represented by an artificial neural network (ANN), under the same criterion
(Goodfellow et al., 2016, Sect. 14.1). A random projection
uses a random matrix
without any training: by the Johnson–Lindenstrauss lemma (JL lemma), the
distances between the feature vectors of the data points are
approximately preserved (Vershynin, 2018). Nonlinear
methods beyond the autoencoder include the manifold-learning
family, among them t-SNE and UMAP, which are used mainly for
visualization (Ghojogh et al., 2023; Meilă and Zhang, 2024).
