covmtx.py, which recomputes what the entry states: the script block by block · open in ColabDictionary of Applied Machine Learning: 114 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here
Updated on 2026-11-04
Readiness 100/100
See also matrix random variable variance eigenvalue decomposition eigenvalue linear regression baseline
The covariance matrix of a random vector consists of the variances of the individual entries on its main diagonal and the covariances between different entries off the diagonal. Together with the mean vector, the covariance matrix collects the second-order statistics of a random vector. For a Gaussian random vector, the mean vector and the covariance matrix determine the linear hypothesis with the smallest risk: the model parameters of this hypothesis and its risk can be read off the entries of the covariance matrix. For a Gaussian random vector, the smallest risk of predicting one entry from the others under the squared error loss is the reciprocal of a diagonal entry of its inverse, the precision matrix.
covmtx.py, which recomputes what the entry states: the script block by block · open in ColabB-dataA warm morning at Krems an der Donau usually comes with a warm afternoon, and the mornings vary less than the afternoons. Fig. 1 shows a dataset of $1827$ daily weather measurements at that station. Each data point represents a day, characterized by the minimum and the maximum temperature as its feature vector ${\mathbf x} = (x_{1}, x_{2})^{\top}$. A compact summary of this dataset is the covariance matrix of a random vector obtained by selecting a data point from the dataset uniformly at random. That summary is the sample covariance matrix of the dataset.
B-covFormally, the covariance
matrix of a random vector
${\mathbf x} \in \mathbb{R}^{d}$ is defined as the
expectation of the outer product of the centered vector
${\mathbf x} - \mathbb{E}\{{\mathbf x}\}$ with itself (if it
exists):
\[\mathbf{C}^{({\mathbf x})} := \mathbb{E} \bigg \{ \big( {\mathbf x} - \mathbb{E} \big\{ {\mathbf x} \big\} \big)
\big({\mathbf x} - \mathbb{E} \big\{ {\mathbf x} \big\} \big)^{\top} \bigg\} \text{.}\]
The entry in row $j$ and column $j'$ is the
covariance of $x_{j}$ and
$x_{j'}$. The diagonal entries
$\big( \mathbf{C}^{({\mathbf x})} \big)_{j,j}$ are
therefore the variances of the individual entries of
${\mathbf x}$.
Fig. 1 also shows an ellipse, obtained as
the level set
$\big\{ {\mathbf x} : \big( {\mathbf x} - \mathbb{E}\{{\mathbf x}\} \big)^{\top} \big( \mathbf{C}^{({\mathbf x})} \big)^{-1} \big( {\mathbf x} - \mathbb{E}\{{\mathbf x}\} \big) = 4 \big\}$
for the covariance matrix $\mathbf{C}^{({\mathbf x})}$ obtained from
the weather measurements. The random vector ${\mathbf x}$
collects the minimum and the maximum temperature of a day
selected uniformly at random, so each data point drawn in
Fig. 1 is a realization of
${\mathbf x}$. The mean vector
$\mathbb{E}\{{\mathbf x}\}$ is the center of the cloud and of the
ellipse.
pythondemos/covmtx.py
Beyond its use as a summary for a dataset, the covariance matrix determines the fundamental limit and the optimal method for predicting one feature from the others. The inverse of an invertible covariance matrix, referred to as the precision matrix (Bishop, 2006, Sect. 2.3.1), answers this task through its diagonal entry $\big( \big( \mathbf{C}^{({\mathbf x})} \big)^{-1} \big)_{j,j}$. The remaining features form the vector ${\mathbf x}_{-j} := \big( x_{1}, \,\ldots, \,x_{j-1}, x_{j+1}, \,\ldots, \,x_{d} \big)^{\top} \in \mathbb{R}^{d-1}$. For a Gaussian random vector ${\mathbf x} \sim \mathcal{N}\left({\boldsymbol \mu},\mathbf{C}^{({\mathbf x})}\right)$ (see multivariate normal distribution), the conditional probability distribution of $x_{j}$, given ${\mathbf x}_{-j}$, is Gaussian with a variance that does not depend on the value of ${\mathbf x}_{-j}$. This conditional variance is $1 \big/ \big( \big( \mathbf{C}^{({\mathbf x})} \big)^{-1} \big)_{j,j}$ (Bishop, 2006, Sect. 2.3.1, Eq. (2.73)).
B-precThis conditional variance becomes a bound on every machine learning (ML) method once the data points can be approximated as realizations of independent and identically distributed (i.i.d.) random vectors ${\mathbf x}^{(r)} \sim \mathcal{N}\left({\boldsymbol \mu},\mathbf{C}^{({\mathbf x})}\right)$. The conditional variance is also the smallest risk that any hypothesis can achieve under the squared error loss, \begin{equation} \label{equ_covmtx_bayesrisk_dict} \min_{h: \mathbb{R}^{d-1} \to \mathbb{R}} \mathbb{E} \Big\{ \big( x_{j} - h\big({\mathbf x}_{-j}\big) \big)^{2} \Big\} = 1 \big/ \big( \big( \mathbf{C}^{({\mathbf x})} \big)^{-1} \big)_{j,j} \text{.} \end{equation} This smallest risk is the Bayes risk of this regression problem. It is attained by the Bayes estimator $\mathbb{E}\{ x_{j} \mid {\mathbf x}_{-j} \}$, which depends linearly on ${\mathbf x}_{-j}$ in the Gaussian case (Bertsekas and Tsitsiklis, 2008, Sect. 8.3; Bishop, 2006, Sect. 2.3.1). For the weather measurements of Fig. 1, the best linear hypothesis predicting the minimum temperature of a day from its maximum temperature has the risk $13.5$ squared degrees Celsius, against the variance $49.9$ of the minimum temperature, which is the risk of the best constant prediction. Its two model parameters are computed from $\mathbf{C}^{({\mathbf x})}$ and $\mathbb{E}\{{\mathbf x}\}$: the slope is the ratio of the covariance to the variance of the maximum temperature, and the intercept is fixed by the two means (see linear regression). Under a Gaussian probabilistic model this risk is also the Bayes risk. By the Bayes risk identity \(\eqref{equ_covmtx_bayesrisk_dict}\), the diagonal of $\big( \mathbf{C}^{({\mathbf x})} \big)^{-1}$ yields a baseline (Friedman et al., 2008).
The spectral decomposition also yields a feature transformation
that removes linear correlations. If $\mathbf{C}^{({\mathbf x})}$ is
invertible, define its inverse square root as
\[
\big(\mathbf{C}^{({\mathbf x})}\big)^{-1/2}
:= \mathbf{V} \boldsymbol{\Lambda}^{-1/2} \mathbf{V}^{\top},
\]
where $\boldsymbol{\Lambda}^{-1/2}$ is obtained by replacing every
eigenvalue with its inverse square root. The transformed
feature vector
\[
\widetilde{{\mathbf x}}
:= \big(\mathbf{C}^{({\mathbf x})}\big)^{-1/2}
\big({\mathbf x}-\mathbb{E}\{{\mathbf x}\}\big)
\]
has zero mean and covariance matrix
$\mathbf{C}^{(\widetilde{{\mathbf x}})}=\mathbf{I}$. Thus, its entries have unit
variance and are mutually uncorrelated, as illustrated in
Fig. 2. This transformation is known as
whitening (Bishop, 2006, Sect. 12.1.3, Eqs. (12.24)–(12.25)).
By removing differences in scale and linear correlations
between the features, whitening can improve the conditioning
of optimization problems and thereby speed up gradient descent (GD) and related
gradient-based methods. In practice, the mean and covariance
matrix used for this transformation are estimated solely from the
training set.
See also: covariance, matrix, random variable, random vector, variance, sample covariance matrix, multivariate normal distribution, spectral decomposition, eigenvalue decomposition, eigenvalue, linear regression, Bayes risk, baseline.
@misc{dictml_covmtx,
author = {Jung, Alexander},
editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
title = {covariance matrix},
howpublished = {Dictionary of Applied Machine Learning (course edition)},
year = {2026},
doi = {10.5281/zenodo.21569296},
note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-04},
url = {https://dictionaryofml.org/terms/covmtx.html}
}