# probability distribution

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-10-31. [HTML](https://dictionaryofml.org/terms/probdist.html) · [typeset PDF](https://dictionaryofml.org/terms/probdist.pdf) · [LaTeX](https://dictionaryofml.org/terms/probdist.tex) · [Python demo](https://dictionaryofml.org/terms/probdist.py) · [notebook](https://dictionaryofml.org/terms/probdist.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A probability distribution of a random variable is the probability measure that the random variable induces on the set of its values: it assigns a probability to every measurable set of values. A machine learning method reads in a dataset and delivers an output, such as a learned hypothesis or the prediction for a given data point, and should work well for any dataset that could arise in the machine learning application at hand. The independent and identically distributed assumption models such a dataset as realizations of independent and identically distributed RVs with a common probability distribution, which is postulated from domain knowledge or estimated from a dataset. The probability distribution of a binary random variable is fully specified by a single probability, that of a discrete random variable by a probability mass function, and that of a real-valued random variable with a probability density function by that probability density function.

## Definition

Consider a machine learning method that takes as input a dataset $\mathcal{D} = \big\{ {\mathbf z}^{(1)}, \,\ldots, \,{\mathbf z}^{(m)} \big\}$, each data point a numeric feature paired with a numeric label, and delivers an output, such as a learned hypothesis $\widehat{h}$ or the prediction for a given data point. Feeding a different dataset into the same method can deliver a different output. In general, the method should work well for any dataset that could arise in the machine learning application at hand. The independent and identically distributed assumption models such a dataset as independent and identically distributed RVs ${\mathbf z}^{(1)}, \,\ldots, \,{\mathbf z}^{(m)} \sim
\mathbb{P}$ with a common probability distribution $\mathbb{P}$ (see Fig. 1). The outputs obtained from different such datasets are themselves realizations of a random variable, whose probability distribution depends on $\mathbb{P}$ and on the method. How much the outputs spread, i.e., how much they differ from one another, is therefore determined by $\mathbb{P}$ and the method.

![Figure 1 of the entry probdist](https://dictionaryofml.org/terms/fig/probdist-fig1.svg)

*Figure 1: Three datasets of realizations of independent and identically distributed RVs drawn from a common probability distribution $\mathbb{P}$, and the three outputs of the same machine learning method $\mathcal{A}$. These three outputs are realizations of another random variable $\widehat{h}$; the probability distribution of $\widehat{h}$ depends on $\mathbb{P}$ and the machine learning method. How much the three outputs differ from one another is therefore determined by $\mathbb{P}$ and the machine learning method*

Formally, a random variable is a measurable function ${\mathbf z}: \Omega \to \mathcal{Z}$ from a probability space $(\Omega,\, \Sigma,\, \mathbb{P})$ to a measurable space $\mathcal{Z}$, the set of values that ${\mathbf z}$ can take. The probability distribution of ${\mathbf z}$ is the probability measure $\mathbb{P}^{({\mathbf z})}$ that ${\mathbf z}$ induces on $\mathcal{Z}$. This probability measure assigns to every measurable subset $\mathcal{B} \subseteq \mathcal{Z}$ the probability

$$
\mathbb{P}^{({\mathbf z})}(\mathcal{B}) :=
\mathbb{P}\left({\mathbf z} \in \mathcal{B}\right) =
\mathbb{P}\big( \{ \omega \in \Omega :
{\mathbf z}(\omega) \in \mathcal{B} \} \big)
\text{,}
$$

the probability of the event that ${\mathbf z}$ takes a value in $\mathcal{B}$ (see Fig. 2; Billingsley, 1986; Gray, 2009; Kallenberg, 1997). That event is the preimage ${\mathbf z}^{-1}(\mathcal{B}) = \{ \omega \in
\Omega : {\mathbf z}(\omega) \in \mathcal{B} \}$, the set of outcomes $\omega$ of the sample space $\Omega$ that ${\mathbf z}$ sends into $\mathcal{B}$. Measurability of ${\mathbf z}$ guarantees that ${\mathbf z}^{-1}(\mathcal{B})$ belongs to the σ-algebra $\Sigma$, so that $\mathbb{P}$ assigns ${\mathbf z}^{-1}(\mathcal{B})$ a probability. The superscript of $\mathbb{P}^{({\mathbf z})}$ names the random variable whose distribution is meant, as in $\mathbb{P}^{({\mathbf x},y)}$ for a data point with features ${\mathbf x}$ and label $y$. When only one random variable is in play, the superscript is dropped, and ${\mathbf z} \sim \mathbb{P}$ states that $\mathbb{P}$ is the probability distribution of ${\mathbf z}$.

![Figure 2 of the entry probdist](https://dictionaryofml.org/terms/fig/probdist-fig2.svg)

*Figure 2: A random variable ${\mathbf z}$ maps each outcome $\omega$ of the sample space $\Omega$ to a value ${\mathbf z}(\omega)$ in the value space $\mathcal{Z}$. Its probability distribution $\mathbb{P}^{({\mathbf z})}$ assigns to a measurable set $\mathcal{B} \subseteq \mathcal{Z}$ (right, shaded) the probability that $\mathbb{P}$ assigns to the preimage ${\mathbf z}^{-1}(\mathcal{B}) \subseteq \Omega$ (left, shaded), the event that ${\mathbf z}$ takes a value in $\mathcal{B}$*

Under the independent and identically distributed assumption, the probability distribution $\mathbb{P}$ of the data points determines how likely each dataset is: a dataset of realizations of independent and identically distributed RVs is likely when its data points lie where $\mathbb{P}$ assigns large probability. This distribution is generally unknown; only the dataset is observed. In practice, $\mathbb{P}$ is postulated based on domain knowledge or estimated from a dataset, e.g., by its empirical distribution or histogram, or by maximum likelihood within a parametric family. A parametric family is a set of probability distributions indexed by a few numbers, such as the normal distributions indexed by mean and variance.

A probability mass function or a probability density function is a compact way of specifying a probability distribution. The probability distribution of a binary random variable $y \in \{0,1\}$ is fully specified by the single probability $\mathbb{P}\left(y = 0\right)$, since $\mathbb{P}\left(y=1\right) = 1-\mathbb{P}\left(y=0\right)$; more generally, the probability distribution of a discrete random variable is specified by its probability mass function. The probability distribution of a real-valued random variable $x \in \mathbb{R}$ that has a probability density function $p^{(x)}\left(\cdot\right)$ is specified by that probability density function. The probability of a value in an interval $[a,b]$ is the integral

$$
\mathbb{P}\left( x \in [a,b] \right) = \int_{a}^{b}
p^{(x)}\left(x\right) \, \mathrm{d}x \text{,}
$$

which is approximately $p^{(x)}\left(a\right) \, |b-a|$ for a short interval $[a,b]$.

Because the probability density function gives the probability of a short interval around each value, it can be drawn. For data points ${\mathbf z} = (x, y)$ with a single numeric feature $x$ and a numeric label $y$, a probability distribution can be visualized in the plane $\mathbb{R}^{2}$. Fig. 3 shows one as a grayscale over the feature–label plane: the darker a region, the larger its probability density function, i.e., the more likely a data point drawn from $\mathbb{P}$ falls in it. The depicted probability density function is a weighted sum of two Gaussian pdfs, i.e., a Gaussian mixture model. Paralleling Fig. 1, three datasets $\mathcal{D}^{(1)}, \mathcal{D}^{(2)}, \mathcal{D}^{(3)}$, each consisting of $m = 20$ realizations of independent and identically distributed RVs with this distribution, are shown together with the three hypotheses $\widehat{h}^{(1)},
\widehat{h}^{(2)}, \widehat{h}^{(3)}$ learned from them by the same polynomial regression method. As shown in Fig. 3, the three learned hypotheses differ most where the probability density function is small.

![Figure 3 of the entry probdist](https://dictionaryofml.org/terms/fig/probdist-fig3.svg)

*Figure 3: A probability distribution over data points ${\mathbf z} = (x, y)$, drawn as a grayscale: the darker a region, the larger its probability density function. The probability density function is a Gaussian mixture model with two components. Paralleling Fig. 1, three datasets $\mathcal{D}^{(1)}$ (filled circles), $\mathcal{D}^{(2)}$ (open squares), and $\mathcal{D}^{(3)}$ (filled triangles), each of $m = 20$ realizations of independent and identically distributed RVs with this distribution, yield the three hypotheses $\widehat{h}^{(1)}$ (solid curve), $\widehat{h}^{(2)}$ (dashed curve), and $\widehat{h}^{(3)}$ (dotted curve), learned by the same polynomial regression method. The hypotheses differ most where the probability density function is small. Data generated by pythondemos/probdist.py*

The example also illustrates what empirical risk minimization can determine from a dataset. Empirical risk minimization chooses the hypothesis by its average loss over the dataset. It therefore gives little weight to a region in which the dataset contains few or no data points. Consequently, empirical risk minimization does not distinguish between hypotheses that differ only there and have nearly the same training error.

**Synonyms:** distribution, law.

**See also:** [independent and identically distributed](https://dictionaryofml.org/terms/iid.html), [independent and identically distributed assumption](https://dictionaryofml.org/terms/iidasspt.html), realization, [random variable](https://dictionaryofml.org/terms/rv.html), probability, measurable, probability density function, probability mass function, discrete random variable, probability measure, probability space, empirical distribution, expectation, [generalization](https://dictionaryofml.org/terms/generalization.html).

## References

- Billingsley (1986). Probability and Measure. *Wiley*.
- Gray (2009). Probability, Random Processes, and Ergodic Properties. *Springer Science+Business Media*. <https://doi.org/10.1007/978-1-4419-1090-5>
- Kallenberg (1997). Foundations of Modern Probability. *Springer-Verlag*.

## Cite this entry

```bibtex
@misc{dictml_probdist,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {probability distribution},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-04},
  url = {https://dictionaryofml.org/terms/probdist.html}
}
```