Dictionary of Applied Machine Learning: 115 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

independent and identically distributed (i.i.d.)

Updated on 2026-11-07

Readiness 100/100

See also random variable independent and identically distributed assumption probability distribution Gaussian random variable covariance matrix variance

Definition

B-diceTwo rolls of the same die are independent and identically distributed: both values follow the same probability distribution, and the first says nothing about the second. A collection of random variables (RVs) with these two properties is called i.i.d. Formally, consider RVs ${\mathbf z}^{(1)}, \ldots, {\mathbf z}^{(m)}$, all defined on one common probability space $(\Omega,\, \Sigma,\, \mathbb{P})$ and taking values in a measurable space $\mathcal{Z}$. They are i.i.d. if every ${\mathbf z}^{(r)}$ follows the same probability distribution and the RVs are mutually independent. The common probability space is what makes mutual independence well defined: one outcome has to fix all $m$ values at once (see RV). The two requirements are separate, and each can fail without the other. Two sensors of different accuracy, read once each, deliver independent values that are not identically distributed. The temperature measured by one sensor at two consecutive times is identically distributed and dependent, since the second value almost repeats the first.

B-drawMutual independence is reflected by a factorization of the probability distribution: the probability of any combination of their values is the product of one probability per RV. For any measurable sets $\mathcal{A}_{1}, \ldots, \mathcal{A}_{m} \subseteq \mathcal{Z}$, \begin{equation} \label{eq:iid_product} \mathbb{P}\left( {\mathbf z}^{(1)} \in \mathcal{A}_{1}, \ldots, {\mathbf z}^{(m)} \in \mathcal{A}_{m}\right) = \prod_{r=1}^{m} \mathbb{P}\left( {\mathbf z}^{(r)} \in \mathcal{A}_{r}\right) \text{.} \end{equation} The product rule \(\eqref{eq:iid_product}\) has to hold for the whole collection, not only for pairs. Let $b_{1}$ and $b_{2}$ be independent RVs, each taking the values $0$ and $1$ with probability $1/2$, and let $b_{3}$ be their sum modulo two. All three take the values $0$ and $1$ with probability $1/2$, so they are identically distributed, and any two of them are independent. The three together are not: $\mathbb{P}\left(b_{1} = 1, b_{2} = 1, b_{3} = 1\right) = 0$, because $1 + 1 = 0$ modulo two, while the product of the three single probabilities is $1/8$. Pairwise independence is therefore weaker than mutual independence, and these three RVs are identically distributed and pairwise independent without being i.i.d. (Gray and Davisson, 2004, Problem 4.31).

B-figFor jointly Gaussian random variables (Gaussian RVs) the two requirements take one compact form. Consider a random vector ${\mathbf x} = (x_{1}, \ldots, x_{d})^{\top}$ of jointly Gaussian RVs with mean vector ${\boldsymbol \mu}$ and covariance matrix ${\mathbf C}$. Its components are i.i.d. exactly when ${\boldsymbol \mu} = (\mu, \ldots, \mu)^{\top}$ and ${\mathbf C} = \sigma^{2} \mathbf{I}$ is a scaled identity matrix. Indeed, for ${\mathbf C} = \sigma^{2} \mathbf{I}$ the joint probability density function (pdf) (Murphy, 2012, Eq. (4.1)) is a product of univariate Gaussian densities with means $\mu_{j}$ and the common variance $\sigma^{2}$, so the components are independent, and they are identically distributed exactly when the means agree. Conversely, i.i.d. components have equal means, equal variances, and zero covariances. Every component then has the same variance $\sigma^{2}$, whose square root $\sigma$ is the standard deviation along each axis. The off-diagonal zeros make the components uncorrelated. For jointly Gaussian RVs this already implies mutual independence (Anderson, 2003, Sect. 2.4), so the three RVs $b_{1}, b_{2}, b_{3}$ have no jointly Gaussian counterpart. The equal diagonal entries and the equal means make the components identically distributed. Fig. 1 shows realizations of such a random vector for $d = 2$.

See also: random variable, independent and identically distributed assumption, probability distribution, probability space, event, measurable, realization, Gaussian random variable, covariance matrix, random vector, variance.

References

1. Gray and Davisson (2004). An Introduction to Statistical Signal Processing. Cambridge Univ. Press. ee.stanford.edu/~gray/sp.pdf
2. Murphy (2012). Machine Learning: A Probabilistic Perspective. MIT Press.
3. Anderson (2003). An Introduction to Multivariate Statistical Analysis. Wiley.

Cite this entry

@misc{dictml_iid,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {independent and identically distributed (i.i.d.)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/iid.html}
}