# independent and identically distributed (i.i.d.)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-07. [HTML](https://dictionaryofml.org/terms/iid.html) · [typeset PDF](https://dictionaryofml.org/terms/iid.pdf) · [LaTeX](https://dictionaryofml.org/terms/iid.tex) · [Python demo](https://dictionaryofml.org/terms/iid.py) · [notebook](https://dictionaryofml.org/terms/iid.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A collection of RVs is independent and identically distributed (i.i.d.) if all of them follow the same probability distribution and they are mutually independent. Two rolls of the same die are i.i.d.: both values have the same probability distribution, and the first says nothing about the second. The two requirements are separate, and each one can fail without the other. Mutual independence means that the probability of any combination of values is the product of one probability per random variable. This is stronger than independence of every pair. For jointly Gaussian RVs the two requirements hold exactly when all means are equal and the covariance matrix is a scaled identity matrix.

## Definition

Two rolls of the same die are independent and identically distributed: both values follow the same probability distribution, and the first says nothing about the second. A collection of RVs with these two properties is called i.i.d. Formally, consider RVs ${\mathbf z}^{(1)}, \ldots, {\mathbf z}^{(m)}$, all defined on one common probability space $(\Omega,\, \Sigma,\, \mathbb{P})$ and taking values in a measurable space $\mathcal{Z}$. They are i.i.d. if every ${\mathbf z}^{(r)}$ follows the same probability distribution and the RVs are mutually independent. The common probability space is what makes mutual independence well defined: one outcome has to fix all $m$ values at once (see random variable). The two requirements are separate, and each can fail without the other. Two sensors of different accuracy, read once each, deliver independent values that are not identically distributed. The temperature measured by one sensor at two consecutive times is identically distributed and dependent, since the second value almost repeats the first.

Mutual independence is reflected by a factorization of the probability distribution: the probability of any combination of their values is the product of one probability per random variable. For any measurable sets $\mathcal{A}_{1}, \ldots, \mathcal{A}_{m} \subseteq
\mathcal{Z}$,

\begin{equation}
\tag{1}
\mathbb{P}\left( {\mathbf z}^{(1)} \in \mathcal{A}_{1}, \ldots,
{\mathbf z}^{(m)} \in \mathcal{A}_{m}\right)
= \prod_{r=1}^{m}
\mathbb{P}\left( {\mathbf z}^{(r)} \in \mathcal{A}_{r}\right)
\text{.}
\end{equation}

The product rule Eq. (1) has to hold for the whole collection, not only for pairs. Let $b_{1}$ and $b_{2}$ be independent RVs, each taking the values $0$ and $1$ with probability $1/2$, and let $b_{3}$ be their sum modulo two. All three take the values $0$ and $1$ with probability $1/2$, so they are identically distributed, and any two of them are independent. The three together are not: $\mathbb{P}\left(b_{1} = 1, b_{2} = 1, b_{3} = 1\right) = 0$, because $1 + 1 = 0$ modulo two, while the product of the three single probabilities is $1/8$. Pairwise independence is therefore weaker than mutual independence, and these three RVs are identically distributed and pairwise independent without being i.i.d. (Gray and Davisson, 2004, Problem 4.31).

For jointly Gaussian RVs the two requirements take one compact form. Consider a random vector ${\mathbf x} = (x_{1}, \ldots, x_{d})^{\top}$ of jointly Gaussian RVs with mean vector ${\boldsymbol \mu}$ and covariance matrix ${\mathbf C}$. Its components are i.i.d. exactly when ${\boldsymbol \mu} = (\mu, \ldots, \mu)^{\top}$ and ${\mathbf C} = \sigma^{2} \mathbf{I}$ is a scaled identity matrix. Indeed, for ${\mathbf C} = \sigma^{2} \mathbf{I}$ the joint probability density function (Murphy, 2012, Eq. (4.1)) is a product of univariate Gaussian densities with means $\mu_{j}$ and the common variance $\sigma^{2}$, so the components are independent, and they are identically distributed exactly when the means agree. Conversely, i.i.d. components have equal means, equal variances, and zero covariances. Every component then has the same variance $\sigma^{2}$, whose square root $\sigma$ is the standard deviation along each axis. The off-diagonal zeros make the components uncorrelated. For jointly Gaussian RVs this already implies mutual independence (Anderson, 2003, Sect. 2.4), so the three RVs $b_{1}, b_{2}, b_{3}$ have no jointly Gaussian counterpart. The equal diagonal entries and the equal means make the components identically distributed. Fig. 1 shows realizations of such a random vector for $d = 2$.

![Figure 1 of the entry iid](https://dictionaryofml.org/terms/fig/iid-fig1.svg)

*Figure 1: Realizations of a pair $x_{1}, x_{2}$ of i.i.d. Gaussian RVs with mean $\mu$ and variance $\sigma^{2}$, each drawn as an open circle. The solid curve has radius $\sigma$ and is centered at $(\mu, \mu)$. Identical probability distributions make the spread the same along both axes, and mutual independence leaves the cloud untilted, so the covariance matrix is $\sigma^{2}$ times the identity matrix. Data generated by pythondemos/iid.py*

**See also:** [random variable](https://dictionaryofml.org/terms/rv.html), [independent and identically distributed assumption](https://dictionaryofml.org/terms/iidasspt.html), [probability distribution](https://dictionaryofml.org/terms/probdist.html), probability space, event, measurable, realization, [Gaussian random variable](https://dictionaryofml.org/terms/gaussrv.html), [covariance matrix](https://dictionaryofml.org/terms/covmtx.html), random vector, [variance](https://dictionaryofml.org/terms/variance.html).

## References

- Gray and Davisson (2004). An Introduction to Statistical Signal Processing. *Cambridge Univ. Press*. <https://ee.stanford.edu/~gray/sp.pdf>
- Murphy (2012). Machine Learning: A Probabilistic Perspective. *MIT Press*.
- Anderson (2003). An Introduction to Multivariate Statistical Analysis. *Wiley*.

## Cite this entry

```bibtex
@misc{dictml_iid,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {independent and identically distributed (i.i.d.)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/iid.html}
}
```