# distribution shift

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-11. [HTML](https://dictionaryofml.org/terms/distshift.html) · [typeset PDF](https://dictionaryofml.org/terms/distshift.pdf) · [LaTeX](https://dictionaryofml.org/terms/distshift.tex) · [Python demo](https://dictionaryofml.org/terms/distshift.py) · [notebook](https://dictionaryofml.org/terms/distshift.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A data point carries features and a label, drawn together from one joint probability distribution. Distribution shift is a mismatch between that joint probability distribution during training and the one the learned hypothesis is applied to after deployment. The average loss on a test set estimates the risk under the training probability distribution, so a small test error is no evidence about the loss incurred after deployment. A joint probability distribution has two components, a marginal distribution and a conditional probability distribution, and the named special cases differ in which of them changes: covariate shift changes the marginal distribution of the features alone, label shift the marginal distribution of the labels alone, and concept drift the conditional probability distribution. Sample weighting corrects covariate shift when both probability distributions are known; online learning applies when they are not.

## Definition

The Finnish Meteorological Institute (FMI) station Helsinki Kaisaniemi records the minimum and the maximum air temperature of each day. Every summer day is a data point: its feature is the minimum temperature of that day, its label the maximum. The generation of these data points can be approximately modeled as RVs with a joint probability distribution $\mathbb{P}$ of feature $x$ and label $y$. A linear hypothesis $\widehat{h}$, learned by linear regression from $62$ of the $92$ data points of June–August 2026, i.e., by minimizing the average squared error loss, reaches a training error of $5.0$ and a test error of $6.5$ on a test set that consists of the remaining $30$ data points.

The learned hypothesis $\widehat{h}$ is then applied to data points $\mathcal{D}'$ collected at the northernmost FMI station, Utsjoki Nuorgam, during December 2025–February 2026. As with the training set, the generation of these data points can also be approximately modeled as RVs with some joint probability distribution $\mathbb{P}'$. The resulting average loss on $\mathcal{D}'$ is $242.1$, which is $37$ times larger than its test error (Fig. 1). The reason for this discrepancy between test error and the performance on $\mathcal{D}'$ is a distribution shift.

![Figure 1 of the entry distshift](https://dictionaryofml.org/terms/fig/distshift-fig1.svg)

*Figure 1: Distribution shift between the data points generated at two different FMI stations. Each marker is one data point, its feature the daily minimum and its label the daily maximum temperature. The markers indicate every sixth data point of the training set (filled circles), every fourth of the test set (open squares), and every fifth after deployment (open diamonds). The line depicts the hypothesis $\widehat{h}$ learned via linear regression from the training set. The features of the data points encountered after deployment lie outside the range covered by the training set. Moreover, the learned hypothesis $\widehat{h}$ overestimates every label after deployment, by $14.6\,^{\circ}$C on average. Data generated by pythondemos/distshift.py*

Distribution shift refers to a discrepancy between the probability distribution $\mathbb{P}$ underlying the generation of the data points used for training and the probability distribution $\mathbb{P}'$ underlying the generation of the data points to which the learned hypothesis is applied after deployment (Quiñonero-Candela et al., 2009). A distribution shift cannot be detected from the test error when the test set is generated by $\mathbb{P}$ as well. The average loss of $\widehat{h}$ on such a test set is an unbiased estimator of the risk $\mathbb{E}_{\mathbb{P}}\!\big\{ L\left({\mathbf z},\widehat{h} \right) \big\}$ under $\mathbb{P}$, which may differ substantially from the risk $\mathbb{E}_{\mathbb{P}'}\!\big\{
L\left({\mathbf z},\widehat{h} \right) \big\}$ under $\mathbb{P}'$.

The risk under $\mathbb{P}'$ is the one that matters after deployment, and no test set drawn from $\mathbb{P}$ measures it. A small test error is therefore no evidence about the loss incurred under $\mathbb{P}' \neq \mathbb{P}$. Consider a hypothesis that predicts by comparing the single feature with a threshold (this is a simple decision tree). The two pdfs in Fig. 2, those of a standard normal distribution $\mathbb{P}$ and of a normal distribution $\mathbb{P}'$ with a larger mean and standard deviation, show the effect of replacing $\mathbb{P}$ by $\mathbb{P}'$ on a threshold chosen under $\mathbb{P}$. The fraction of data points beyond the threshold rises from $14\%$ to $88\%$, so the risk under $\mathbb{P}'$ exceeds the risk under $\mathbb{P}$ that the test error estimates.

![Figure 2 of the entry distshift](https://dictionaryofml.org/terms/fig/distshift-fig2.svg)

*Figure 2: Distribution shift for a single feature. The solid curve is the probability density function of the probability distribution $\mathbb{P}$ that generated the training set, here a standard normal distribution, with mean $0$ and standard deviation $1$; the dashed curve is the probability density function of the probability distribution $\mathbb{P}'$ at deployment, a normal distribution with mean $2.5$ and standard deviation $1.2$. The dotted threshold of the learned hypothesis $\widehat{h}$ at $x = 1.1$ divides each probability density function into two parts; the area to its right is the fraction of data points whose feature exceeds the threshold, $14\%$ under $\mathbb{P}$ against $88\%$ under $\mathbb{P}'$, both computed from the normal distribution*

Three named special cases of distribution shift correspond to different origins of the shift from $\mathbb{P}$ to $\mathbb{P}'$. These different origins correspond to the factors in the two factorizations of the joint probability distribution. Both hold for any joint probability distribution of a numeric feature vector and label, whether or not a probability density function exists, since the conditional probability distribution is then guaranteed to exist (Billingsley, 1986, Thm. 33.3; Kallenberg, 1997, Thms. 5.3 and 5.4):

$$
\mathbb{P}^{({\mathbf x}, y)}
= \mathbb{P}^{({\mathbf x})} \cdot
\mathbb{P}^{(y \mid {\mathbf x})}
= \mathbb{P}^{(y)} \cdot
\mathbb{P}^{({\mathbf x} \mid y)} \text{.}
$$

Covariate shift and label shift change the marginal distributions $\mathbb{P}^{({\mathbf x})}$ of the feature vector and $\mathbb{P}^{(y)}$ of the label, respectively. Concept drift refers to a change in the conditional probability distribution $\mathbb{P}^{(y \mid {\mathbf x})}$ (with $\mathbb{P}^{({\mathbf x})}$ unchanged) or $\mathbb{P}^{({\mathbf x} \mid y)}$ (with $\mathbb{P}^{(y)}$ unchanged).

The two-station example from the beginning is not a pure case of any of the three. The marginal distribution $\mathbb{P}^{({\mathbf x})}$ changes, since the winter data points have lower feature values. The conditional $\mathbb{P}^{(y \mid {\mathbf x})}$ changes as well: a linear hypothesis learned from the $90$ Nuorgam winter data points has slope $0.77$, against the slope $0.48$ of the hypothesis learned at Helsinki, so a data point whose feature is one degree higher has a label $0.77$ degrees higher in winter, against $0.48$ in summer. The class-conditional $\mathbb{P}^{({\mathbf x} \mid y)}$ changes too. In particular, the two datasets have no label value in common, since every label at Nuorgam in winter is at most $4.6\,^{\circ}$C and every label at Helsinki in summer at least $15.0\,^{\circ}$C.

Distribution shift is detected by comparing summary statistics of the features between the training set and the data points arriving after deployment, or by monitoring the incurred loss whenever labels become available (Quiñonero-Candela et al., 2009).

One mitigation of distribution shift is sample weighting which can be used when the probability distribution $\mathbb{P}$ and $\mathbb{P}'$ are known. Here, with $p$ and $p'$ the pdfs of $\mathbb{P}$ and $\mathbb{P}'$, each data point of the training set is weighted by the ratio $p'({\mathbf x})/p({\mathbf x})$. This weighting corrects covariate shift under two conditions: $\mathbb{P}^{(y \mid {\mathbf x})}$ is unchanged, and every feature vector occurring under $\mathbb{P}'$ also occurs under $\mathbb{P}$, so that the ratio is defined (Quiñonero-Candela et al., 2009). Under these two conditions (Shimodaira, 2000),

$$
p'({\mathbf x}, y) = p(y \mid {\mathbf x})\,
p'({\mathbf x}) =
\big(p'({\mathbf x})/p({\mathbf x})\big)\,
p({\mathbf x}, y) \text{, wherever } p({\mathbf x}) > 0 \text{.}
$$

Thus, the weighted loss has the same expectation under $\mathbb{P}$ as the loss has under $\mathbb{P}'$.

Another approach to mitigate distribution shift is to use online learning, which updates the hypothesis as data points from the changing probability distribution arrive (Quiñonero-Candela et al., 2009). It needs their labels to become available after deployment, which sample weighting does not, and in return it needs neither probability distribution to be known.

**Synonyms:** dataset shift, domain shift.

**See also:** [generalization](https://dictionaryofml.org/terms/generalization.html), [independent and identically distributed assumption](https://dictionaryofml.org/terms/iidasspt.html), [probability distribution](https://dictionaryofml.org/terms/probdist.html), risk, [training set](https://dictionaryofml.org/terms/trainset.html), [test set](https://dictionaryofml.org/terms/testset.html), covariate shift, [transfer learning](https://dictionaryofml.org/terms/transferlearning.html), domain adaptation, sample weighting, [online learning](https://dictionaryofml.org/terms/onlinelearning.html).

## References

- Quiñonero-Candela et al. (2009). Dataset Shift in Machine Learning. *MIT Press*.
- Billingsley (1986). Probability and Measure. *Wiley*.
- Kallenberg (1997). Foundations of Modern Probability. *Springer-Verlag*.
- Shimodaira (2000). Improving predictive inference under covariate shift by weighting the log-likelihood function. *J. Statist. Plann. Inference*. <https://doi.org/10.1016/S0378-3758(00)00115-4>

## Cite this entry

```bibtex
@misc{dictml_distshift,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {distribution shift},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/distshift.html}
}
```