A matrix is a two-dimensional array of numbers,
arranged in rows and columns. The entry $A_{r,j}$ of a
matrix ${\mathbf A} \in \mathbb{R}^{m \times d}$ sits
in row $r$ and column $j$. A prominent
example in machine learning (ML) is the feature matrix of a dataset,
obtained by stacking the feature vectors of $m$
data points row-wise. A matrix represents several distinct
mathematical objects. One is a system of linear equations,
such as the normal equations of linear regression. Another is a
linear map between two vector spaces, after fixing a
basis for each. A third is a tabular dataset with one
row per data point and one column per feature.
P-featuremtxConsider a spreadsheet of daily weather
records at one station: each row holds a single day, each column
one measurement such as temperature or humidity. Such a tabular
dataset holds $m$ data points, each described
by the same number $d$ of numerical features. The features of
the $r$-th data point are collected in its
feature vector
${\mathbf x}^{(r)} = \big(x^{(r)}_{1},
\ldots, x^{(r)}_{d}\big)^{\top} \in
\mathbb{R}^{d}$, one entry per column of the spreadsheet.
Stacking these $m$ feature vectors row-wise, so that
the $r$-th row is
$\big({\mathbf x}^{(r)}\big)^{\top}$, produces an
$m \times d$ rectangular array of numbers. Its
entry in row $r$ and column $j$ is the
$j$-th feature of the $r$-th
data point. This two-dimensional numeric array is
referred to as the feature matrix ${\mathbf X}$ of the
dataset; Fig. 1 carries out the
construction on three days of the weather example.
Figure 1: A tabular dataset of $\samplesize = 3$
data points and the feature matrix built from it. Each
row of the table is one day, whose $\nrfeatures = 3$
measurements, temperature (temp.), humidity (humid.), and
pressure (press.), form the feature vector
$\featurevec^{(\sampleidx)} \in \reals^{3}$; the day column
indexes the data points and is not itself a feature,
so it does not appear in $\featuremtx$. Stacking
$\featurevec^{(1)}, \featurevec^{(2)}, \featurevec^{(3)}$ as rows
makes the entry $X_{\sampleidx,\featureidx}$ the $\featureidx$-th
measurement taken on day $\sampleidx$: $X_{2,3} = 1008$ is the
pressure on day $2$
More generally, a matrix of size $m \times d$
is a two-dimensional array of numbers denoted by
\[
{\mathbf A} = \begin{pmatrix}
A_{1,1} & A_{1,2} & \dots & A_{1,d} \\
A_{2,1} & A_{2,2} & \dots & A_{2,d} \\
\vdots & \vdots & \ddots & \vdots \\
A_{m,1} & A_{m,2} & \dots & A_{m,d}
\end{pmatrix} \in \mathbb{R}^{m \times d}\text{.}
\]
Here, $A_{r,j}$ denotes the matrix entry
in the $r$-th row and the $j$-th column.
A digital image is a matrix of exactly this
kind. Recording how green each pixel is turns a $16 \times 16$ image
into a $16 \times 16$ matrix $\mathbf{G}$ whose two indices are pixel
positions (Fig. 2).
Figure 2: A $16 \times 16$ pixel image of the "for all" quantifier
$\forall$ and the matrix of its green channel. Pixel
$(\sampleidx,\featureidx)$ has greenness $G_{\sampleidx,
\featureidx} = 1$ where the glyph is drawn and $0$ elsewhere, so
the $1$s trace the same shape in $\mG$ that the shaded squares
trace in the image
Matrices represent several distinct mathematical objects
(Strang, 2016), including the following:
Systems of linear equations: A matrix collects the
coefficients of a system of linear equations,
\[ \begin{pmatrix}
A_{1,1} & A_{1,2} \\
A_{2,1} & A_{2,2}
\end{pmatrix}
\begin{pmatrix}
x_1 \\
x_2
\end{pmatrix}
=\begin{pmatrix}
b_1 \\
b_2
\end{pmatrix}
\text{, written compactly as } {\mathbf A} {\mathbf x} = {\mathbf b} \text{.}
\]
Here, ${\mathbf x}$ is the vector of unknowns and ${\mathbf b}$ the
right-hand side. One important example is the normal equations
${\mathbf X}^{\top}{\mathbf X}\,{\mathbf w} = {\mathbf X}^{\top}{\mathbf y}$,
whose solutions are the model parameters that minimize the
training error in linear regression, as the derivation following
Fig. 3 shows.
Linear maps:
Consider two vector spaces $\mathcal{U}$ and $\mathcal{V}$, of dimension $d$ and $m$, respectively.
Fixing a basis ${\mathbf u}^{(1)},\,\ldots,\,{\mathbf u}^{(d)}$ for $\mathcal{U}$ and a basis ${\mathbf v}^{(1)},\,\ldots,\,{\mathbf v}^{(m)}$
for $\mathcal{V}$, each matrix ${\mathbf A} \in \mathbb{R}^{m \times d}$ defines a
linear map $f^{({\mathbf A})}: \mathcal{U} \rightarrow \mathcal{V}$ (see Fig. 3) via
\[f^{({\mathbf A})}\big({\mathbf u}^{(j)}\big) = \sum_{r=1}^{m} A_{r,j} {\mathbf v}^{(r)}\text{.}\]
Taking $\mathcal{U} = \mathbb{R}^{d}$ and
$\mathcal{V} = \mathbb{R}^{m}$ with the standard bases,
the image of a vector ${\mathbf x}$ under $f^{({\mathbf A})}$ is the
matrix-vector product $f^{({\mathbf A})}({\mathbf x}) = {\mathbf A}{\mathbf x}$.
Datasets: As in the opening example, a matrix
can represent a dataset with each row holding a
single data point and each column a specific
feature or label.
Figure 3: The matrix $\mA$ defines a linear map $f^{(\mA)}$ between two
vector spaces: the basis vector $\vu^{(\featureidx)}$
is mapped to the vector (thick red arrows) whose
coordinates form the
$\featureidx$-th column of $\mA$
Written with the feature matrix, the
training error that empirical risk minimization (ERM)-based linear regression minimizes
becomes a succinct algebraic expression. Collecting the labels of the
training set in ${\mathbf y} \in \mathbb{R}^{m}$, the
training error of the hypothesis
$h^{({\mathbf w})}({\mathbf x}) := {\mathbf w}^{\top}{\mathbf x}$
is
\[
(1/m)\,\left\lVert {\mathbf X} {\mathbf w} - {\mathbf y} \right\rVert_{2}^{2}
\text{,}
\]
with gradient
$(2/m)\,{\mathbf X}^{\top}({\mathbf X} {\mathbf w} - {\mathbf y})$.
Each gradient descent (GD) step is therefore one multiplication by ${\mathbf X}$
followed by one by ${\mathbf X}^{\top}$, and equating the
gradient to zero leads back to the normal equations
from the first example above.
Matrices appear across machine learning (ML) beyond the feature matrix. The
Jacobian matrix and the Hessian collect the first- and
second-order partial derivatives of a multivariate
function and are the input of gradient-based training. The
covariance matrix records the pairwise covariances of a
probability distribution over features. Principal component analysis (PCA), singular value decomposition (SVD), and a
random projection factor or compress the feature matrix to
lower its dimension. A kernel method replaces explicit
feature vectors by a matrix of pairwise inner products, and
a tabular Markov decision process (MDP) stores its transition probabilities as a matrix.
A graph is represented by matrices as well: its
adjacency matrix lists which nodes are joined and with what
edge weight. The Laplacian matrix assembled from those weights
(Luxburg, 2007, Sect. 3) is the matrix whose
eigenvectors graph clustering uses to split the nodes
into clusters.
See also: linear map, dataset, linear model,
vector, feature matrix, normal equations,
array, transpose.
References
Strang (2016). Introduction to Linear Algebra. Wellesley-Cambridge Press.
