probability distribution

Jung, Alexander

Dictionary of Applied Machine Learning: 114 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

probability distribution

Updated on 2026-10-31

Readiness 100/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Markdown Cite this entry

A probability distribution of a random variable (RV) is the probability measure that the RV induces on the set of its values: it assigns a probability to every measurable set of values. A machine learning (ML) method reads in a dataset and delivers an output, such as a learned hypothesis or the prediction for a given data point, and should work well for any dataset that could arise in the ML application at hand. The independent and identically distributed assumption (i.i.d. assumption) models such a dataset as realizations of independent and identically distributed (i.i.d.) RVs with a common probability distribution, which is postulated from domain knowledge or estimated from a dataset. The probability distribution of a binary RV is fully specified by a single probability, that of a discrete random variable (discrete RV) by a probability mass function (pmf), and that of a real-valued RV with a probability density function (pdf) by that pdf.

Figure produced by the Python demo of the entry probability distribution — Figure written by the entry's Python demo `probdist.py`, which recomputes what the entry states: the script block by block · open in Colab

Definition

P-methodConsider a machine learning (ML) method that takes as input a dataset $\mathcal{D} = \big\{ {\mathbf z}^{(1)}, \,\ldots, \,{\mathbf z}^{(m)} \big\}$, each data point a numeric feature paired with a numeric label, and delivers an output, such as a learned hypothesis $\widehat{h}$ or the prediction for a given data point. Feeding a different dataset into the same method can deliver a different output. In general, the method should work well for any dataset that could arise in the ML application at hand. The independent and identically distributed assumption (i.i.d. assumption) models such a dataset as independent and identically distributed (i.i.d.) random variables (RVs) ${\mathbf z}^{(1)}, \,\ldots, \,{\mathbf z}^{(m)} \sim \mathbb{P}$ with a common probability distribution $\mathbb{P}$ (see Fig. 1). The outputs obtained from different such datasets are themselves realizations of an RV, whose probability distribution depends on $\mathbb{P}$ and on the method. How much the outputs spread, i.e., how much they differ from one another, is therefore determined by $\mathbb{P}$ and the method.

Figure 1 of the entry probdist — Figure 1: Three datasets of realizations of i.i.d. RVs drawn from a common probability distribution $\probdist$, and the three outputs of the same ML method $\algomap$. These three outputs are realizations of another RV $\learnthypothesis$; the probability distribution of $\learnthypothesis$ depends on $\probdist$ and the ML method. How much the three outputs differ from one another is therefore determined by $\probdist$ and the ML method

Formally, an RV is a measurable function ${\mathbf z}: \Omega \to \mathcal{Z}$ from a probability space $(\Omega,\, \Sigma,\, \mathbb{P})$ to a measurable space $\mathcal{Z}$, the set of values that ${\mathbf z}$ can take. The probability distribution of ${\mathbf z}$ is the probability measure $\mathbb{P}^{({\mathbf z})}$ that ${\mathbf z}$ induces on $\mathcal{Z}$. This probability measure assigns to every measurable subset $\mathcal{B} \subseteq \mathcal{Z}$ the probability \[ \mathbb{P}^{({\mathbf z})}(\mathcal{B}) := \mathbb{P}\left({\mathbf z} \in \mathcal{B}\right) = \mathbb{P}\big( \{ \omega \in \Omega : {\mathbf z}(\omega) \in \mathcal{B} \} \big) \text{,} \] the probability of the event that ${\mathbf z}$ takes a value in $\mathcal{B}$ (see Fig. 2; Billingsley, 1986; Gray, 2009; Kallenberg, 1997). That event is the preimage ${\mathbf z}^{-1}(\mathcal{B}) = \{ \omega \in \Omega : {\mathbf z}(\omega) \in \mathcal{B} \}$, the set of outcomes $\omega$ of the sample space $\Omega$ that ${\mathbf z}$ sends into $\mathcal{B}$. Measurability of ${\mathbf z}$ guarantees that ${\mathbf z}^{-1}(\mathcal{B})$ belongs to the $\sigma$-algebra $\Sigma$, so that $\mathbb{P}$ assigns ${\mathbf z}^{-1}(\mathcal{B})$ a probability. The superscript of $\mathbb{P}^{({\mathbf z})}$ names the RV whose distribution is meant, as in $\mathbb{P}^{({\mathbf x},y)}$ for a data point with features ${\mathbf x}$ and label $y$. When only one RV is in play, the superscript is dropped, and ${\mathbf z} \sim \mathbb{P}$ states that $\mathbb{P}$ is the probability distribution of ${\mathbf z}$.

Figure 2 of the entry probdist — Figure 2: An RV $\datapoint$ maps each outcome $\outcome$ of the sample space $\samplespace$ to a value $\datapoint(\outcome)$ in the value space $\datapointspace$. Its probability distribution $\probdist^{(\datapoint)}$ assigns to a measurable set $\mathcal{B} \subseteq \datapointspace$ (right, shaded) the probability that $\probdist$ assigns to the preimage $\datapoint^{-1}(\mathcal{B}) \subseteq \samplespace$ (left, shaded), the event that $\datapoint$ takes a value in $\mathcal{B}$

Under the i.i.d. assumption, the probability distribution $\mathbb{P}$ of the data points determines how likely each dataset is: a dataset of realizations of i.i.d. RVs is likely when its data points lie where $\mathbb{P}$ assigns large probability. This distribution is generally unknown; only the dataset is observed. In practice, $\mathbb{P}$ is postulated based on domain knowledge or estimated from a dataset, e.g., by its empirical distribution or histogram, or by maximum likelihood within a parametric family. A parametric family is a set of probability distributions indexed by a few numbers, such as the normal distributions indexed by mean and variance.

P-specifyA probability mass function (pmf) or a probability density function (pdf) is a compact way of specifying a probability distribution. The probability distribution of a binary RV $y \in \{0,1\}$ is fully specified by the single probability $\mathbb{P}\left(y = 0\right)$, since $\mathbb{P}\left(y=1\right) = 1-\mathbb{P}\left(y=0\right)$; more generally, the probability distribution of a discrete random variable (discrete RV) is specified by its pmf. The probability distribution of a real-valued RV $x \in \mathbb{R}$ that has a pdf $p^{(x)}\left(\cdot\right)$ is specified by that pdf. The probability of a value in an interval $[a,b]$ is the integral \[ \mathbb{P}\left( x \in [a,b] \right) = \int_{a}^{b} p^{(x)}\left(x\right) \, \mathrm{d}x \text{,} \] which is approximately $p^{(x)}\left(a\right) \, |b-a|$ for a short interval $[a,b]$.

P-pictureBecause the pdf gives the probability of a short interval around each value, it can be drawn. For data points ${\mathbf z} = (x, y)$ with a single numeric feature $x$ and a numeric label $y$, a probability distribution can be visualized in the plane $\mathbb{R}^{2}$. Fig. 3 shows one as a grayscale over the feature–label plane: the darker a region, the larger its pdf, i.e., the more likely a data point drawn from $\mathbb{P}$ falls in it. The depicted pdf is a weighted sum of two Gaussian pdfs, i.e., a Gaussian mixture model (GMM). Paralleling Fig. 1, three datasets $\mathcal{D}^{(1)}, \mathcal{D}^{(2)}, \mathcal{D}^{(3)}$, each consisting of $m = 20$ realizations of i.i.d. RVs with this distribution, are shown together with the three hypotheses $\widehat{h}^{(1)}, \widehat{h}^{(2)}, \widehat{h}^{(3)}$ learned from them by the same polynomial regression method. As shown in Fig. 3, the three learned hypotheses differ most where the pdf is small.

Figure 3 of the entry probdist — Figure 3: A probability distribution over data points $\datapoint = (\feature, \truelabel)$, drawn as a grayscale: the darker a region, the larger its pdf. The pdf is a GMM with two components. Paralleling Fig. 1, three datasets $\dataset^{(1)}$ (filled circles), $\dataset^{(2)}$ (open squares), and $\dataset^{(3)}$ (filled triangles), each of $\samplesize = 20$ realizations of i.i.d. RVs with this distribution, yield the three hypotheses $\learnthypothesis^{(1)}$ (solid curve), $\learnthypothesis^{(2)}$ (dashed curve), and $\learnthypothesis^{(3)}$ (dotted curve), learned by the same polynomial regression method. The hypotheses differ most where the pdf is small. Data generated by `pythondemos/probdist.py`

The example also illustrates what empirical risk minimization (ERM) can determine from a dataset. ERM chooses the hypothesis by its average loss over the dataset. It therefore gives little weight to a region in which the dataset contains few or no data points. Consequently, ERM does not distinguish between hypotheses that differ only there and have nearly the same training error.

Synonyms: distribution, law.

See also: independent and identically distributed, independent and identically distributed assumption, realization, random variable, probability, measurable, probability density function, probability mass function, discrete random variable, probability measure, probability space, empirical distribution, expectation, generalization.

References

Billingsley (1986). Probability and Measure. Wiley.
Gray (2009). Probability, Random Processes, and Ergodic Properties. Springer Science+Business Media. doi.org/10.1007/978-1-4419-1090-5
Kallenberg (1997). Foundations of Modern Probability. Springer-Verlag.

Cite this entry

@misc{dictml_probdist,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {probability distribution},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-04},
  url = {https://dictionaryofml.org/terms/probdist.html}
}