# correlation

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-01. [HTML](https://dictionaryofml.org/terms/correlation.html) · [typeset PDF](https://dictionaryofml.org/terms/correlation.pdf) · [LaTeX](https://dictionaryofml.org/terms/correlation.tex) · [Python demo](https://dictionaryofml.org/terms/correlation.py) · [notebook](https://dictionaryofml.org/terms/correlation.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Correlation measures the strength and direction of the linear relationship between two real-valued RVs. It is obtained by normalizing their covariance by their standard deviations, making it dimensionless and comparable across quantities measured on different scales. Its extreme positive and negative values characterize perfect increasing and decreasing affine relationships, while its central value characterizes uncorrelated variables. Independent RVs are uncorrelated, but uncorrelated RVs need not be independent, since correlation quantifies only linear dependence. The correlation between the predictions of the base learners of an ensemble bounds the variance reduction that averaging them can achieve.

## Definition

Fig. 1 shows a dataset that consists of $1827$ daily weather measurements at Krems an der Donau. Each data point represents a day, characterized by the minimum and the maximum temperature as its feature vector ${\mathbf x}=(x_{1},x_{2})^{\top}$. Selecting a data point from the dataset uniformly at random defines a random vector ${\mathbf x}$. Correlation quantifies whether a day whose minimum temperature is unusually high also tends to have an unusually high maximum temperature.

More generally, for two real-valued RVs $x$ and $y$ on a common probability space with nonzero standard deviations $\sigma_{x}$ and $\sigma_{y}$, their correlation is their covariance normalized by the product of these standard deviations (Bertsekas and Tsitsiklis, 2008),

$$
\rho := \frac{{\rm cov}\left(x,y\right)}{\sigma_{x}\,\sigma_{y}} \in [-1, 1]\text{.}
$$

The value $\rho$ is also referred to as the Pearson correlation coefficient.

Unlike the covariance, correlation is dimensionless. Adding a constant to either random variable, or multiplying it by a positive constant, does not change $\rho$; multiplying one random variable by a negative constant reverses its sign (Anderson, 2003, Sec. 2.3, Thm. 2.3.2). The extreme values $\rho = 1$ and $\rho = -1$ occur exactly when $y$ is an increasing or a decreasing affine function of $x$ with probability one (Meyer, 2000, Sec. 5.4). The RVs are uncorrelated when $\rho = 0$ (Bertsekas and Tsitsiklis, 2008, Sec. 4.2). Independent RVs are uncorrelated, but uncorrelated RVs need not be independent, since $\rho$ quantifies only linear dependence (Bertsekas and Tsitsiklis, 2008, Sec. 4.2). For the random vector ${\mathbf x}$ obtained by selecting uniformly at random one of the daily minimum-and-maximum temperature pairs shown in Fig. 1, the covariance between the two temperatures is $55.08$ squared degrees Celsius, and their variances are $49.89$ and $83.41$ squared degrees Celsius. Thus,

$$
\rho = \frac{55.08}{\sqrt{49.89\cdot83.41}} \approx 0.85\text{.}
$$

![Figure 1 of the entry correlation](https://dictionaryofml.org/terms/fig/correlation-fig1.svg)

*Figure 1: Daily realizations of the weather random vector ${\mathbf x}$: every sixth day from $2020$ to $2024$. The correlation $\rho\approx0.85$, computed from all $1827$ days, reflects the tendency of high daily minima to accompany high daily maxima. Data generated by pythondemos/correlation.py*

The correlation between the predictions of the base learners of an ensemble limits how much averaging them can help. An average of $M$ identically distributed predictions, each with variance $\sigma^{2}$ and pairwise correlation $\rho$, has variance $\rho \sigma^{2} + (1 - \rho)\sigma^{2} / M$ (Hastie et al., 2009, Sect. 15.2). The second term vanishes as $M$ grows, while the first remains, which is why bagging resamples the training set and a random forest randomizes the features at each split: both lower the correlation between their predictions.

## References

- Bertsekas and Tsitsiklis (2008). Introduction to Probability. *Athena Scientific*.
- Anderson (2003). An Introduction to Multivariate Statistical Analysis. *Wiley*.
- Meyer (2000). Matrix Analysis and Applied Linear Algebra. *SIAM*.
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>

## Cite this entry

```bibtex
@misc{dictml_correlation,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {correlation},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-02},
  url = {https://dictionaryofml.org/terms/correlation.html}
}
```