\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath,amssymb}
\usepackage{algorithm,algpseudocode}
\usepackage{bm}
\usepackage{hyperref}
\usepackage{float}
\usepackage{subcaption}
\usepackage{enumitem}
\usepackage{xcolor}
\definecolor{darkgreen}{rgb}{0.0, 0.5, 0.0}
\definecolor{aaltoBlue}{RGB}{0, 51, 102}
\definecolor{pinegreen}{cmyk}{0.92,0,0.59,0.25}
\definecolor{royalblue}{cmyk}{1,0.50,0,0}
\definecolor{lavander}{cmyk}{0,0.48,0,0}
\definecolor{violet}{cmyk}{0.79,0.88,0,0}
\definecolor{lightblue}{RGB}{173, 216, 230}
\usepackage{tikz}
\usepackage{pgfplots}
\usetikzlibrary{calc,arrows.meta,positioning,fit,shapes.geometric,shapes.misc,intersections,patterns,decorations.pathreplacing,backgrounds,matrix}
\usepgfplotslibrary{fillbetween,groupplots}
\setcounter{secnumdepth}{-\maxdimen}
% Long unbreakable tokens (the ISO/IEC author names in citations, term names
% in the title) otherwise overflow the margin by a few points; a modest
% emergency stretch lets TeX loosen the paragraph instead.
\emergencystretch=2em
\usepackage{mleftright}
% Project macros used by this entry (auto-extracted).
\newcommand{\covmtxgeneric}{\mC}
\newcommand{\datapoint}{\vz}
\newcommand{\datapointspace}{\mathcal{Z}}
\newcommand{\feature}{x}
\newcommand{\featuredim}{d}
\newcommand{\featurevec}{\vx}
\newcommand{\genericevent}{\mathcal{A}}
\newcommand{\mC}[0]{{\bf C}}
\newcommand{\mI}{\mathbf{I}}
\newcommand{\meanvecgeneric}{{\bm \mu}}
\newcommand{\prob}[1]{\mathbb{P}\left(#1\right)}
\newcommand{\probdist}{\mathbb{P}}
\newcommand{\probspace}{(\samplespace,\, \sigmaalgebra,\, \probdist)}
\newcommand{\sampleidx}{r}
\newcommand{\samplesize}{m}
\newcommand{\samplespace}{\Omega}
\newcommand{\sigmaalgebra}{\Sigma}
\newcommand{\vx}[0]{{\bf x}}
\newcommand{\vz}[0]{{\bf z}}

\begin{document}

\section{independent and identically distributed (i.i.d.)}

\subsection*{Author and editors}
Alexander Jung\textsuperscript{*}, Department of Computer Science, Aalto University, Espoo, Finland (ORCID \mbox{\href{https://orcid.org/0000-0001-7538-0990}{0000-0001-7538-0990}}) \\
Editors: Konstantina Olioumtsevits (ORCID \mbox{\href{https://orcid.org/0000-0003-2697-8659}{0000-0003-2697-8659}}), Ekkehard Schnoor (ORCID \mbox{\href{https://orcid.org/0000-0001-8252-1499}{0000-0001-8252-1499}}), Aalto University, Espoo, Finland \\
\textsuperscript{*}Corresponding author: \href{mailto:alex.jung@aalto.fi}{alex.jung@aalto.fi}



\subsection{Abstract}
A collection of random variables (RVs) is independent and identically
		distributed (i.i.d.) if all of them follow the same probability distribution
		and they are mutually independent. Two rolls of the same die are
		i.i.d.: both values have the same probability distribution, and the first
		says nothing about the second. The two requirements are separate,
		and each one can fail without the other. Mutual independence means
		that the probability of any combination of values is the
		product of one probability per RV. This is stronger
		than independence of every pair. For jointly Gaussian random variables (Gaussian RVs) the
		two requirements hold exactly when all means are equal and
		the covariance matrix is a scaled identity matrix.

\subsection{Definition}
Two
		rolls of the same die are independent and identically distributed:
		both values follow the same probability distribution, and the first says
		nothing about the second. A collection of random variables (RVs) with these two
		properties is called i.i.d. Formally, consider RVs
		$\datapoint^{(1)}, \ldots, \datapoint^{(\samplesize)}$, all defined
		on one common probability space $\probspace$ and taking values in a
		measurable space $\datapointspace$. They are i.i.d.\ if every
		$\datapoint^{(\sampleidx)}$ follows the same probability distribution and the
		RVs are mutually independent. The common probability space
		is what makes mutual independence well defined: one outcome
		has to fix all $\samplesize$ values at once (see RV).
		The two requirements are separate, and each can fail without the
		other. Two sensors of different accuracy, read once each, deliver
		independent values that are not identically distributed. The
		temperature measured by one sensor at two consecutive times is
		identically distributed and dependent, since the second value
		almost repeats the first.
\par
		Mutual independence is reflected by a factorization of the
		probability distribution: the probability of any combination of their
		values is the product of one probability per RV. For
		any measurable sets
		$\genericevent_{1}, \ldots, \genericevent_{\samplesize} \subseteq
		\datapointspace$,
		\begin{equation}
		\label{eq:iid_product}
		\prob{ \datapoint^{(1)} \in \genericevent_{1}, \ldots,
			\datapoint^{(\samplesize)} \in \genericevent_{\samplesize}}
		= \prod_{\sampleidx=1}^{\samplesize}
		\prob{ \datapoint^{(\sampleidx)} \in \genericevent_{\sampleidx}}
		\text{.}
		\end{equation}
		The product rule \eqref{eq:iid_product} has to hold for the whole
		collection, not only for pairs. Let $b_{1}$ and $b_{2}$ be
		independent RVs, each taking the values $0$ and $1$ with
		probability $1/2$, and let $b_{3}$ be their sum modulo two.
		All three take the values $0$ and $1$ with probability $1/2$,
		so they are identically distributed, and any two of them are
		independent. The three together are not:
		$\prob{b_{1} = 1, b_{2} = 1, b_{3} = 1} = 0$, because
		$1 + 1 = 0$ modulo two, while the product of the three single
		probabilities is $1/8$. Pairwise independence is therefore
		weaker than mutual independence, and these three RVs are
		identically distributed and pairwise independent without being
		i.i.d.\ (Gray and Davisson, 2004, Problem~4.31).
\par
		For jointly Gaussian random variables (Gaussian RVs) the two requirements take one compact
		form. Consider a random vector
		$\featurevec = (\feature_{1}, \ldots, \feature_{\featuredim})^{\top}$
		of jointly Gaussian RVs with mean vector
		$\meanvecgeneric$ and covariance matrix $\covmtxgeneric$. Its components are
		i.i.d.\ exactly when
		$\meanvecgeneric = (\mu, \ldots, \mu)^{\top}$ and
		$\covmtxgeneric = \sigma^{2} \mI$ is a scaled identity matrix.
		Indeed, for $\covmtxgeneric = \sigma^{2} \mI$ the joint
		probability density function (pdf) (Murphy, 2012, Eq.~(4.1)) is a product of
		univariate Gaussian densities with means $\mu_{j}$ and the
		common variance $\sigma^{2}$, so the components are
		independent, and they are identically distributed exactly when
		the means agree. Conversely, i.i.d.\ components have equal
		means, equal variances, and zero
		covariances. Every component then has the same
		variance $\sigma^{2}$, whose square root $\sigma$ is the
		standard deviation along each axis. The off-diagonal zeros make the
		components uncorrelated. For jointly Gaussian RVs this already
		implies mutual independence
		(Anderson, 2003, Sect.~2.4), so the
		three RVs $b_{1}, b_{2}, b_{3}$ have no jointly Gaussian
		counterpart. The equal diagonal entries and the equal means
		make the components identically distributed.
		Fig.~\ref{fig_iid_scatter_dict} shows realizations of such a
		random vector for $\featuredim = 2$.
		\begin{figure}[H]
		\centering
		\begin{tikzpicture}
			\begin{axis}[
				width=0.62\linewidth, axis equal image,
				xlabel={$\feature_{1}$}, ylabel={$\feature_{2}$},
				axis lines=left,
				xmin=-4.5, xmax=8.5, ymin=-4.5, ymax=8.5,
				tick label style={font=\small},
				label style={font=\small},
				legend style={draw=none, fill=none, font=\scriptsize,
					at={(0.5,-0.22)}, anchor=north},
				]
				\addplot[only marks, mark=o, mark size=1.1pt, black!55]
					table[col sep=comma, x=x1, y=x2]
					{iid_points.csv};
				\addlegendentry{realization of the pair
					$(\feature_{1}, \feature_{2})$}
				\addplot[thick, black, no marks]
					table[col sep=comma, x=x1, y=x2]
					{iid_circle.csv};
				\addlegendentry{circle of radius $\sigma$ around
					$(\mu, \mu)$}
			\end{axis}
		\end{tikzpicture}
		\caption{Realizations of a pair
			$\feature_{1}, \feature_{2}$ of i.i.d.\ Gaussian RVs with
			mean $\mu$ and variance $\sigma^{2}$, each drawn as
			an open circle. The solid curve has radius $\sigma$ and is
			centered at $(\mu, \mu)$. Identical
			probability distributions make the spread the same along both axes, and
			mutual independence leaves the cloud untilted, so the
			covariance matrix is $\sigma^{2}$ times the identity matrix.
			Data generated by \texttt{pythondemos/iid.py}
			\label{fig_iid_scatter_dict}}
		\end{figure}

\subsection{Cross References}
RV, i.i.d.\ assumption, probability distribution, probability space, event, measurable, realization, Gaussian RV, covariance matrix, random vector, variance

\subsection{References}
\begin{enumerate}
  \item Anderson (2003). An Introduction to Multivariate Statistical Analysis. 3rd ed., Wiley, Hoboken, NJ.
  \item Gray and Davisson (2004). An Introduction to Statistical Signal Processing. Cambridge Univ. Press, Cambridge, U.K.
  \item Murphy (2012). Machine Learning: A Probabilistic Perspective. MIT Press, Cambridge, MA, USA.
\end{enumerate}

\end{document}