Dictionary of Applied Machine Learning

probability distribution

Updated on 2026-09-17

Typeset PDF version — the authoritative form of this entry

Python demo — a script that recomputes what this entry states and prints one line per check

A probability distribution assigns probabilities to the possible values of a random variable (RV). A machine learning (ML) method reads in a dataset and delivers an output, such as a learned hypothesis or the prediction for a given data point, and should work well for any dataset that is typical for the ML application at hand. One widely used construction of a typical dataset is as realizations of independent and identically distributed (i.i.d.) RVs with a common probability distribution, postulated from domain knowledge or estimated from a dataset. The probability distribution of a binary RV is fully specified by a single probability, that of a continuous real-valued RV may be specified by a probability density function (pdf), and in the most general case by a probability measure.

Definition

Consider a machine learning (ML) method that reads in a dataset $\dataset = \big\{ \datapoint^{(1)}, \,\ldots, \,\datapoint^{(\samplesize)} \big\}$ and delivers an output, such as a learned hypothesis $\learnthypothesis$ or the prediction for a given data point. Feeding a different dataset into the same method can deliver a different output. In general, the method should work well for any dataset that is typical for the ML application at hand. One widely used construction of a typical dataset is as independent and identically distributed (i.i.d.) random variables (RVs) $\datapoint^{(1)}, \,\ldots, \,\datapoint^{(\samplesize)} \sim \probdist$ with a common probability distribution $\probdist$ (see Fig. 1).

P-typicalThe probability distribution $\probdist$ decides which sequences of data points (or datasets) are typical. The distribution $\probdist$ is never observed; only the dataset is. In practice, $\probdist$ can be postulated based on domain knowledge or estimated from a dataset, e.g., by its empirical distribution or histogram, or by maximum likelihood within a parametric family.

P-pictureHow a probability distribution is specified depends on the values the RV takes. The probability distribution of a binary RV $\truelabel \in \{0,1\}$ is fully specified by the single probability $\prob{\truelabel = 0}$, since $\prob{\truelabel=1} = 1-\prob{\truelabel=0}$; more generally, an RV with finitely many values is specified by a probability mass function (pmf). The probability distribution of a continuous real-valued RV $\feature \in \reals$ might be specified by a probability density function (pdf) $\pdf{\feature}{\cdot}$, for which $\prob{ \feature \in [a,b] } \approx \pdf{\feature}{a} \, |b-a|$ holds for a short interval $[a,b]$. In the most general case, an RV is a map $\datapoint: \samplespace \to \datapointspace$ from a probability space $(\samplespace, \sigmaalgebra, \prob{\cdot})$ to its value space $\datapointspace$. The probability distribution of $\datapoint$ is the measure that $\datapoint$ induces on $\datapointspace$: it assigns to a measurable subset $\mathcal{B} \subseteq \datapointspace$ the probability $\prob{\datapoint \in \mathcal{B}}$ (Billingsley, 1986; Gray, 2009; Kallenberg, 1997).

Figure 1 of the entry probdist
Figure 1: Three datasets of realizations of i.i.d. RVs drawn from a common probability distribution $\probdist$, and the three outputs of the same ML method $\algomap$. These three outputs are realizations of another RV $\learnthypothesis$; the probability distribution of $\learnthypothesis$ depends on $\probdist$ and the ML method. The spread of the outputs is a property of $\probdist$
For data points $\datapoint = (\feature, \truelabel)$ with a single numeric feature $\feature$ and a numeric label $\truelabel$, a probability distribution can be visualized in the plane $\reals^{2}$. Fig. 2 shows one as a grayscale over the feature–label plane: the darker a region, the larger its pdf, and the more typical the data points in it. The depicted pdf is a weighted sum of two Gaussian pdfs, i.e., a Gaussian mixture model (GMM). Paralleling Fig. 1, three datasets $\dataset_{1}, \dataset_{2}, \dataset_{3}$, each consisting of $\samplesize = 40$ realizations of i.i.d. RVs with this distribution, are shown together with the three hypothesis maps $\learnthypothesis_{1}, \learnthypothesis_{2}, \learnthypothesis_{3}$ learned from them by the same polynomial regression method. The three learned maps differ most where the pdf is small: a region that the datasets rarely cover constrains the learned hypotheses only weakly.
Figure 2 of the entry probdist
Figure 2: A probability distribution over data points $\datapoint = (\feature, \truelabel)$, drawn as a grayscale: the darker a region, the larger its pdf. The pdf is a GMM with two components. Paralleling Fig. 1, three datasets $\dataset_{1}, \dataset_{2}, \dataset_{3}$ of $\samplesize = 40$ realizations of i.i.d. RVs with this distribution (three marker shapes) yield the three hypothesis maps $\learnthypothesis_{1}, \learnthypothesis_{2}, \learnthypothesis_{3}$ learned by the same polynomial regression method (three line styles). The maps differ most where the pdf is small. Data generated by pythondemos/probdist.py
Synonyms: distribution, law.

See also: independent and identically distributed, realization, random variable, probability, probability density function, probability mass function, probability measure, probability space, empirical distribution, expectation, generalization.

References

  1. Billingsley (1986). Probability and Measure. Wiley.
  2. Gray (2009). Probability, Random Processes, and Ergodic Properties. Springer Science+Business Media. doi.org/10.1007/978-1-4419-1090-5
  3. Kallenberg (1997). Foundations of Modern Probability. Springer-Verlag.

Cite this entry

@misc{dictml_probdist,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {probability distribution},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-09-21},
  url = {https://dictionaryofml.org/terms/probdist.html}
}