B-fig writes when the script runsDictionary of Applied Machine Learning · independent and identically distributed (i.i.d.)
Updated on 2026-11-03
Numerical companion to the entry independent and identically distributed (i.i.d.): it recomputes what the entry states and prints one line per check
Numerical companion to the glossary entry 'iid'. Two real-valued random variables x_1 and x_2 are drawn independently, each from the same Gaussian probability distribution with mean mu and variance sigma^2. Repeating the draw many times gives a cloud of pairs (x_1, x_2).
Run it without installing anything:uv run https://dictionaryofml.org/terms/iid.py
uv downloads this script and the pinned NumPy and Matplotlib it needs, then runs it; the script fetches any input file it uses. To keep the output files, download iid.py into a folder and run uv run iid.py there. With NumPy and Matplotlib already installed, python3 iid.py, from any directory — it writes its output files into the current directory. Fixed seeds, so the printed numbers reproduce exactly. Download iid.py · Notebook · Open in Colab
Before running a downloaded copy, scan it: uvx bandit -q -t B102,B307,B602,B605 iid.py (B102,B307,B602,B605 are the exec, eval and shell checks of bandit) reports nothing: the script runs no shell command and evaluates no generated code.
One cell per block of the script: the code, and what that code printed when it last ran here
"""Realizations of two independent and identically distributed Gaussian
random variables, and the covariance matrix they produce.
Purpose
-------
Numerical companion to the glossary entry 'iid'. Two real-valued random
variables x_1 and x_2 are drawn independently, each from the same
Gaussian probability distribution with mean mu and variance sigma^2.
Repeating the draw many times gives a cloud of pairs (x_1, x_2).
The cloud shows both requirements at once. Identically distributed
makes the spread the same along the two axes, and independent makes the
cloud show no tilt, so the covariance matrix estimated from the
realizations is close to sigma^2 times the identity matrix. A pair that
failed either requirement would be visible: unequal variances stretch
the cloud along one axis, and dependence tilts it.
The demo checks the claims the entry makes. (0) For the two rolls of a
die the entry opens on, both values follow the same distribution and the
joint values factorize into the product of the two single ones, which is
the factorization that mutual independence reflects. (1) The average of
the
drawn pairs is close to (mu, mu). (2) The covariance matrix estimated
from the realizations is close to sigma^2 times the identity: both
diagonal entries are near sigma^2 and the off-diagonal entry is near
zero, within the error expected from the number of draws. (3) The
spread along the two diagonal directions matches sigma^2 as well, which
is the circular shape a scaled identity means. (4) A
deliberately correlated pair with the same two marginals fails the same
test, so the off-diagonal entry is what the independence requirement
controls.
Deterministic: the draws use a fixed seed. Self-contained: numpy +
matplotlib only.
Blocks
------
[B-dice] Two rolls of the same die, many times over: both follow the
same distribution and the joint values factorize into the
product of the two single ones.
[B-draw] Draw the pairs from one Gaussian probability distribution.
[B-moments] The average and the covariance matrix estimated from the
realizations, against mu and sigma^2 times the identity.
[B-tilt] The same two marginals made dependent, for contrast.
[B-fig] The preview figure: the cloud with its one-standard-deviation
circle, and the dependent cloud beside it.
Outputs
-------
iid_points.csv : the drawn pairs
iid_circle.csv : the circle of radius sigma around (mu, mu)
iid_tilted.csv : the dependent pairs of the contrast
iid.png : preview figure
"""
# /// script
# requires-python = ">=3.10"
# dependencies = ["matplotlib==3.10.7", "numpy==2.3.4"]
# ///
from pathlib import Path
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt # noqa: E402
import numpy as np # noqa: E402
OUT_DIR = Path(__file__).parent
MU, SIGMA, NR_DRAWS = 2.0, 1.5, 400
RNG = np.random.default_rng(20240101)
RNG_DICE = np.random.default_rng(7) # its own stream, so the
# Gaussian draws below do not
# depend on the roll count
FAILED = []
def check(name, ok):
FAILED.append(name) if not ok else None
print(f" [{'ok' if ok else 'FAIL'}] {name}")
Two rolls of the same die, many times over: both follow the same distribution and the joint values factorize into the product of the two single ones.
ROLLS = 200_000
d1 = RNG_DICE.integers(1, 7, ROLLS)
d2 = RNG_DICE.integers(1, 7, ROLLS)
joint = np.zeros((6, 6))
for a, b in zip(d1, d2):
joint[a - 1, b - 1] += 1.0
joint /= ROLLS
marg1, marg2 = joint.sum(axis=1), joint.sum(axis=0)
worst = float(np.max(np.abs(joint - np.outer(marg1, marg2))))
print(f"[B-dice] two rolls of the same die, {ROLLS} times: the pair (6, 6) "
f"came up in {joint[5, 5]:.5f} of them against {1 / 36:.5f}")
print(f"[B-dice] over all 36 pairs the largest gap between the joint and "
f"the product of the two single values is {worst:.5f}")
check("[B-dice] the two rolls follow the same distribution, each value "
"within 0.005 of one sixth",
np.all(np.abs(marg1 - 1 / 6) < 0.005)
and np.all(np.abs(marg2 - 1 / 6) < 0.005))
check("[B-dice] and the joint values factorize into the product of the "
"two single ones, so the rolls are independent", worst < 0.005)
[B-dice] two rolls of the same die, 200000 times: the pair (6, 6) came up in 0.02742 of them against 0.02778 [B-dice] over all 36 pairs the largest gap between the joint and the product of the two single values is 0.00078 [ok] [B-dice] the two rolls follow the same distribution, each value within 0.005 of one sixth [ok] [B-dice] and the joint values factorize into the product of the two single ones, so the rolls are independent
Draw the pairs from one Gaussian probability distribution.
X = MU + SIGMA * RNG.standard_normal((NR_DRAWS, 2))
print(f"[B-draw] {NR_DRAWS} pairs, each coordinate drawn from the Gaussian "
f"with mean {MU} and variance {SIGMA ** 2:.2f}")
[B-draw] 400 pairs, each coordinate drawn from the Gaussian with mean 2.0 and variance 2.25
The average and the covariance matrix estimated from the realizations, against mu and sigma^2 times the identity.
avg = X.mean(axis=0)
cov = np.cov(X, rowvar=False)
# the error to expect on each entry from this many draws
band = 3.0 * SIGMA ** 2 / np.sqrt(NR_DRAWS)
print(f"[B-moments] average ({avg[0]:.3f}, {avg[1]:.3f}) against "
f"({MU}, {MU})")
print(f"[B-moments] covariance matrix [[{cov[0,0]:.3f}, {cov[0,1]:.3f}], "
f"[{cov[1,0]:.3f}, {cov[1,1]:.3f}]] against {SIGMA ** 2:.2f} times "
f"the identity")
check("[B-moments] the average of the pairs is close to (mu, mu)",
np.all(np.abs(avg - MU) < 3.0 * SIGMA / np.sqrt(NR_DRAWS)))
check("[B-moments] both diagonal entries are close to sigma squared",
abs(cov[0, 0] - SIGMA ** 2) < band and abs(cov[1, 1] - SIGMA ** 2) < band)
check("[B-moments] the off-diagonal entry is close to zero",
abs(cov[0, 1]) < band)
# a scaled identity means the same spread in EVERY direction, not just
# along the two axes: the two diagonal directions must match as well
diag = np.array([[1.0, 1.0], [1.0, -1.0]]) / np.sqrt(2.0)
spread = (X @ diag.T).var(axis=0, ddof=1)
print(f"[B-moments] spread along the two diagonal directions "
f"{spread[0]:.3f} and {spread[1]:.3f}, also near {SIGMA ** 2:.2f}")
check("[B-moments] the spread along the diagonal directions matches sigma "
"squared too, so the cloud is close to circular",
np.all(np.abs(spread - SIGMA ** 2) < band))
[B-moments] average (2.109, 2.043) against (2.0, 2.0) [B-moments] covariance matrix [[2.305, 0.105], [0.105, 2.340]] against 2.25 times the identity [ok] [B-moments] the average of the pairs is close to (mu, mu) [ok] [B-moments] both diagonal entries are close to sigma squared [ok] [B-moments] the off-diagonal entry is close to zero [B-moments] spread along the two diagonal directions 2.428 and 2.218, also near 2.25 [ok] [B-moments] the spread along the diagonal directions matches sigma squared too, so the cloud is close to circular
The same two marginals made dependent, for contrast.
rho = 0.8
base = RNG.standard_normal((NR_DRAWS, 2))
tilted = np.empty_like(base)
tilted[:, 0] = MU + SIGMA * base[:, 0]
tilted[:, 1] = MU + SIGMA * (rho * base[:, 0]
+ np.sqrt(1.0 - rho ** 2) * base[:, 1])
cov_t = np.cov(tilted, rowvar=False)
print(f"[B-tilt] with dependence the off-diagonal entry is "
f"{cov_t[0,1]:.3f}, not {cov[0,1]:.3f}")
check("[B-tilt] the dependent pair keeps both marginal variances",
abs(cov_t[0, 0] - SIGMA ** 2) < band
and abs(cov_t[1, 1] - SIGMA ** 2) < band)
check("[B-tilt] but its off-diagonal entry sits at rho times sigma "
"squared, far from the zero that independence gives, so the "
"covariance matrix is no longer a multiple of the identity",
abs(cov_t[0, 1] - rho * SIGMA ** 2) < band
and abs(cov_t[0, 1]) > 5 * band)
# ---- the figure data
angle = np.linspace(0.0, 2.0 * np.pi, 256)
circle = np.stack([MU + SIGMA * np.cos(angle), MU + SIGMA * np.sin(angle)], 1)
for name, arr, head in (("iid_points.csv", X, "x1,x2"),
("iid_circle.csv", circle, "x1,x2"),
("iid_tilted.csv", tilted, "x1,x2")):
with open(OUT_DIR / name, "w") as f:
f.write(head + "\n")
for a, b in arr:
f.write(f"{a:.4f},{b:.4f}\n")
[B-tilt] with dependence the off-diagonal entry is 1.842, not 0.105 [ok] [B-tilt] the dependent pair keeps both marginal variances [ok] [B-tilt] but its off-diagonal entry sits at rho times sigma squared, far from the zero that independence gives, so the covariance matrix is no longer a multiple of the identity
The preview figure: the cloud with its one-standard-deviation circle, and the dependent cloud beside it.
fig, (axI, axT) = plt.subplots(1, 2, figsize=(8.6, 4.2))
for ax, pts, title in (
(axI, X, "independent and identically distributed"),
(axT, tilted, "dependent, same two marginals")):
ax.plot(pts[:, 0], pts[:, 1], "o", ms=3, mfc="none", mec="0.25")
ax.plot(circle[:, 0], circle[:, 1], "k-", lw=1.2,
label="circle of radius $\\sigma$ around $(\\mu, \\mu)$")
ax.set_xlabel("$x_1$")
ax.set_ylabel("$x_2$")
ax.set_title(title, fontsize=10)
ax.set_aspect("equal")
ax.set_xlim(MU - 4.2 * SIGMA, MU + 4.2 * SIGMA)
ax.set_ylim(MU - 4.2 * SIGMA, MU + 4.2 * SIGMA)
axI.legend(frameon=False, fontsize=8, loc="upper left")
fig.tight_layout()
fig.savefig(OUT_DIR / "iid.png", dpi=110)
check("[B-fig] the preview figure was written", (OUT_DIR / "iid.png").exists())
print()
if FAILED:
print(f"{len(FAILED)} check(s) FAILED: " + "; ".join(FAILED))
raise SystemExit(1)
print("all checks passed")
[ok] [B-fig] the preview figure was written all checks passed
B-fig writes when the script runs