# covariance matrix

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-04. [HTML](https://dictionaryofml.org/terms/covmtx.html) · [typeset PDF](https://dictionaryofml.org/terms/covmtx.pdf) · [LaTeX](https://dictionaryofml.org/terms/covmtx.tex) · [Python demo](https://dictionaryofml.org/terms/covmtx.py) · [notebook](https://dictionaryofml.org/terms/covmtx.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

The covariance matrix of a random vector consists of the variances of the individual entries on its main diagonal and the covariances between different entries off the diagonal. Together with the mean vector, the covariance matrix collects the second-order statistics of a random vector. For a Gaussian random vector, the mean vector and the covariance matrix determine the linear hypothesis with the smallest risk: the model parameters of this hypothesis and its risk can be read off the entries of the covariance matrix. For a Gaussian random vector, the smallest risk of predicting one entry from the others under the squared error loss is the reciprocal of a diagonal entry of its inverse, the precision matrix.

## Definition

A warm morning at Krems an der Donau usually comes with a warm afternoon, and the mornings vary less than the afternoons. Fig. 1 shows a dataset of $1827$ daily weather measurements at that station. Each data point represents a day, characterized by the minimum and the maximum temperature as its feature vector ${\mathbf x} = (x_{1}, x_{2})^{\top}$. A compact summary of this dataset is the covariance matrix of a random vector obtained by selecting a data point from the dataset uniformly at random. That summary is the sample covariance matrix of the dataset.

Formally, the covariance matrix of a random vector ${\mathbf x} \in \mathbb{R}^{d}$ is defined as the expectation of the outer product of the centered vector ${\mathbf x} - \mathbb{E}\{{\mathbf x}\}$ with itself (if it exists):

$$
\mathbf{C}^{({\mathbf x})} := \mathbb{E} \bigg \{ \big( {\mathbf x} - \mathbb{E} \big\{ {\mathbf x} \big\} \big)
\big({\mathbf x} - \mathbb{E} \big\{ {\mathbf x} \big\} \big)^{\top} \bigg\} \text{.}
$$

The entry in row $j$ and column $j'$ is the covariance of $x_{j}$ and $x_{j'}$. The diagonal entries $\big( \mathbf{C}^{({\mathbf x})} \big)_{j,j}$ are therefore the variances of the individual entries of ${\mathbf x}$. Fig. 1 also shows an ellipse, obtained as the level set $\big\{ {\mathbf x} : \big( {\mathbf x} - \mathbb{E}\{{\mathbf x}\} \big)^{\top} \big( \mathbf{C}^{({\mathbf x})} \big)^{-1} \big( {\mathbf x} - \mathbb{E}\{{\mathbf x}\} \big) = 4 \big\}$ for the covariance matrix $\mathbf{C}^{({\mathbf x})}$ obtained from the weather measurements. The random vector ${\mathbf x}$ collects the minimum and the maximum temperature of a day selected uniformly at random, so each data point drawn in Fig. 1 is a realization of ${\mathbf x}$. The mean vector $\mathbb{E}\{{\mathbf x}\}$ is the center of the cloud and of the ellipse.

![Figure 1 of the entry covmtx](https://dictionaryofml.org/terms/fig/covmtx-fig1.svg)

*Figure 1: (a) Daily temperature measurements at Krems an der Donau, which can be interpreted as realizations of a random vector ${\mathbf x} \in \mathbb{R}^{2}$: every sixth day of the years $2020$ to $2024$ (gray dots), the mean vector $\mathbb{E}\{{\mathbf x}\}$ of all $1827$ days (cross), and the level set at the value four of the quadratic form in $\big( \mathbf{C}^{({\mathbf x})} \big)^{-1}$ (solid curve). The curve is tilted because $x_{1}$ and $x_{2}$ have a positive covariance. (b) The covariance matrix of all $1827$ days, in squared degrees Celsius: the shaded diagonal entries are the variances of $x_{1}$ and $x_{2}$, and the dashed off-diagonal entries are their covariance. Data generated by pythondemos/covmtx.py*

A covariance matrix is always positive semi-definite, i.e., ${\mathbf v}^{\top} \mathbf{C}^{({\mathbf x})} {\mathbf v} \geq 0$ for every vector ${\mathbf v} \in \mathbb{R}^{d}$, since this quadratic form is the variance of ${\mathbf v}^{\top} {\mathbf x}$. Since it is also symmetric, it admits the spectral decomposition

$$
\mathbf{C}^{({\mathbf x})} = \mathbf{V} \boldsymbol{\Lambda} \mathbf{V}^{\top},
$$

where the columns of $\mathbf{V}$ are orthonormal eigenvectors and $\boldsymbol{\Lambda}$ is diagonal with nonnegative eigenvalues. It is invertible, and then positive definite, exactly when all these eigenvalues are positive. Equivalently, no entry of ${\mathbf x}$ is, with probability one, an affine function of the other entries.

Beyond its use as a summary for a dataset, the covariance matrix determines the fundamental limit and the optimal method for predicting one feature from the others. The inverse of an invertible covariance matrix, referred to as the precision matrix (Bishop, 2006, Sect. 2.3.1), answers this task through its diagonal entry $\big( \big( \mathbf{C}^{({\mathbf x})} \big)^{-1} \big)_{j,j}$. The remaining features form the vector ${\mathbf x}_{-j} := \big( x_{1}, \,\ldots, \,x_{j-1}, x_{j+1}, \,\ldots, \,x_{d} \big)^{\top} \in \mathbb{R}^{d-1}$. For a Gaussian random vector ${\mathbf x} \sim \mathcal{N}\left({\boldsymbol \mu},\mathbf{C}^{({\mathbf x})}\right)$ (see multivariate normal distribution), the conditional probability distribution of $x_{j}$, given ${\mathbf x}_{-j}$, is Gaussian with a variance that does not depend on the value of ${\mathbf x}_{-j}$. This conditional variance is $1 \big/ \big( \big( \mathbf{C}^{({\mathbf x})} \big)^{-1} \big)_{j,j}$ (Bishop, 2006, Sect. 2.3.1, Eq. (2.73)).

This conditional variance becomes a bound on every machine learning method once the data points can be approximated as realizations of independent and identically distributed random vectors ${\mathbf x}^{(r)} \sim \mathcal{N}\left({\boldsymbol \mu},\mathbf{C}^{({\mathbf x})}\right)$. The conditional variance is also the smallest risk that any hypothesis can achieve under the squared error loss,

\begin{equation}
\tag{1}
\min_{h: \mathbb{R}^{d-1} \to \mathbb{R}}
\mathbb{E} \Big\{ \big( x_{j}
- h\big({\mathbf x}_{-j}\big) \big)^{2} \Big\}
= 1 \big/ \big( \big( \mathbf{C}^{({\mathbf x})} \big)^{-1}
\big)_{j,j} \text{.}
\end{equation}

This smallest risk is the Bayes risk of this regression problem. It is attained by the Bayes estimator $\mathbb{E}\{ x_{j} \mid {\mathbf x}_{-j} \}$, which depends linearly on ${\mathbf x}_{-j}$ in the Gaussian case (Bertsekas and Tsitsiklis, 2008, Sect. 8.3; Bishop, 2006, Sect. 2.3.1). For the weather measurements of Fig. 1, the best linear hypothesis predicting the minimum temperature of a day from its maximum temperature has the risk $13.5$ squared degrees Celsius, against the variance $49.9$ of the minimum temperature, which is the risk of the best constant prediction. Its two model parameters are computed from $\mathbf{C}^{({\mathbf x})}$ and $\mathbb{E}\{{\mathbf x}\}$: the slope is the ratio of the covariance to the variance of the maximum temperature, and the intercept is fixed by the two means (see linear regression). Under a Gaussian probabilistic model this risk is also the Bayes risk. By the Bayes risk identity Eq. (1), the diagonal of $\big( \mathbf{C}^{({\mathbf x})} \big)^{-1}$ yields a baseline (Friedman et al., 2008).

The spectral decomposition also yields a feature transformation that removes linear correlations. If $\mathbf{C}^{({\mathbf x})}$ is invertible, define its inverse square root as

$$
\big(\mathbf{C}^{({\mathbf x})}\big)^{-1/2}
:= \mathbf{V} \boldsymbol{\Lambda}^{-1/2} \mathbf{V}^{\top},
$$

where $\boldsymbol{\Lambda}^{-1/2}$ is obtained by replacing every eigenvalue with its inverse square root. The transformed feature vector

$$
\widetilde{{\mathbf x}}
:= \big(\mathbf{C}^{({\mathbf x})}\big)^{-1/2}
\big({\mathbf x}-\mathbb{E}\{{\mathbf x}\}\big)
$$

has zero mean and covariance matrix $\mathbf{C}^{(\widetilde{{\mathbf x}})}=\mathbf{I}$. Thus, its entries have unit variance and are mutually uncorrelated, as illustrated in Fig. 2. This transformation is known as whitening (Bishop, 2006, Sect. 12.1.3, Eqs. (12.24)–(12.25)). By removing differences in scale and linear correlations between the features, whitening can improve the conditioning of optimization problems and thereby speed up gradient descent and related gradient-based methods. In practice, the mean and covariance matrix used for this transformation are estimated solely from the training set.

![Figure 2 of the entry covmtx](https://dictionaryofml.org/terms/fig/covmtx-fig2.svg)

*Figure 2: The weather data of Fig. 1 before (left) and after (right) multiplication of each centered feature vector by $\big(\mathbf{C}^{({\mathbf x})}\big)^{-1/2}$. The covariance ellipse becomes a circle, and the transformed features have unit variance and zero covariance.*

**See also:** covariance, [matrix](https://dictionaryofml.org/terms/matrix.html), [random variable](https://dictionaryofml.org/terms/rv.html), random vector, [variance](https://dictionaryofml.org/terms/variance.html), sample covariance matrix, multivariate normal distribution, spectral decomposition, [eigenvalue decomposition](https://dictionaryofml.org/terms/evd.html), [eigenvalue](https://dictionaryofml.org/terms/eigenvalue.html), [linear regression](https://dictionaryofml.org/terms/linreg.html), Bayes risk, [baseline](https://dictionaryofml.org/terms/baseline.html).

## References

- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Bertsekas and Tsitsiklis (2008). Introduction to Probability. *Athena Scientific*.
- Friedman et al. (2008). Sparse inverse covariance estimation with the graphical lasso. *Biostatistics*. <https://doi.org/10.1093/biostatistics/kxm045>

## Cite this entry

```bibtex
@misc{dictml_covmtx,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {covariance matrix},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-04},
  url = {https://dictionaryofml.org/terms/covmtx.html}
}
```