# data augmentation

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-07. [HTML](https://dictionaryofml.org/terms/dataaug.html) · [typeset PDF](https://dictionaryofml.org/terms/dataaug.pdf) · [LaTeX](https://dictionaryofml.org/terms/dataaug.tex) · [Python demo](https://dictionaryofml.org/terms/dataaug.py) · [notebook](https://dictionaryofml.org/terms/dataaug.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

Data augmentation enlarges a training set with synthetic data points obtained from transformations or perturbations of existing ones. Typical examples are rotating or cropping an image and adding small perturbations to the features and labels of a measured data point. If the label does not change under a transformation, training on transformed copies makes the learned hypothesis vary little under it. Sensing devices measure with limited precision, so perturbed copies reflect the uncertainty interval around each recorded value. Data augmentation is a form of regularization, beside model pruning and adding a penalty term to the training error. For linear regression, perturbing the features with zero-mean noise whose covariance matrix is a multiple of the identity is, in expectation over the noise, equivalent to adding the ridge regression penalty term.

## Definition

Three measured days at Krems an der Donau are all a forecaster has to learn from. Each day is a data point, characterized by the maximum daytime temperature as its feature $x$, and the maximum daytime temperature of the next day as its label $y$ (Fig. 1). The figure also shows a learned hypothesis map $\widehat{h}$, obtained by empirical risk minimization using the polynomials of degree two as hypothesis space.

The learned hypothesis achieves zero training error but delivers poor predictions outside the training set, which is overfitting. On a held-out set of $125$ data points obtained from other days of 2024, whose feature lies inside the measured range, its average squared error loss is $256.2$ squared degrees. One reason for the failure of $\widehat{h}$ is that the training set is too small. For almost every training set of sample size $m = 3$ there is one polynomial of degree two passing through its data points.

Data augmentation enlarges a dataset with synthetic data points, obtained by perturbations of the features and labels of the measured ones. Sensing devices have imperfections such as thermal noise, so a reading $x$ is one representative of an uncertainty interval $\mathcal{U}^{(x)}$. Any other value from that interval could have been observed instead.

Considering the weather data from Fig. 1, a temperature reading $x$ carries $\mathcal{U}^{(x)} = [x - 1, x + 1]$. Each measured day $\left( x,y \right)$ is replaced by $400$ copies $\left( x + \varepsilon,y + \varepsilon' \right)$, whose perturbations $\varepsilon$ and $\varepsilon'$ are realizations of RVs distributed uniformly on $[-1, 1]$. The augmented training set $\widetilde{\mathcal{D}}$ holds $1200$ data points, none of which required a new temperature measurement. The width of the uncertainty interval used for the perturbation of the features and labels is a hyperparameter. One basic way to determine it is to try a few choices and compare the validation error of the resulting learned hypotheses.

Fig. 1 also shows the polynomial of degree two $\widehat{h}'$ that empirical risk minimization learns using $\widetilde{\mathcal{D}}$ as training set. The hypothesis $\widehat{h}'$ no longer passes through the three measured days, but it delivers better predictions outside the training set. Its training error on them is $2.8$ squared degrees, but its average squared error loss on the held-out data points falls to $24.7$. The coefficient of $x^{2}$ drops from $\widehat{w}_{2} = 0.71$ for $\widehat{h}$ to $\widehat{w}_{2}' = 0.06$ for $\widehat{h}'$.

![Figure 1 of the entry dataaug](https://dictionaryofml.org/terms/fig/dataaug-fig1.svg)

*Figure 1: Three data points (filled circles) that represent daily weather conditions at Krems an der Donau. The solid curve is the degree-two polynomial $\widehat{h}$ with a training error of zero on them. It dips far below every temperature that was measured. The open squares are copies drawn from the uncertainty interval of each measured day, five of the copies of every day. The dashed curve is the hypothesis $\widehat{h}'$ learned from them. Data generated by pythondemos/dataaug.py*

Enlarging $\mathcal{D}$ with perturbed copies is one basic form of regularization, beside model pruning, which shrinks the hypothesis space, and adding a penalty term to the training error. These different forms of regularization are sometimes equivalent in their effect on the learned hypothesis (Fig. 2). For linear regression, adding zero-mean perturbations with covariance matrix $\sigma^{2} \mathbf{I}$ to each feature vector yields the ridge regression solution, asymptotically in the number of copies. Its penalty term is $\sigma^{2} \left\lVert {\mathbf w} \right\rVert_{2}^{2}$ for the weight vector ${\mathbf w}$ (Bishop, 1995; Goodfellow et al., 2016, Sect. 7.5).

![Figure 2 of the entry dataaug](https://dictionaryofml.org/terms/fig/dataaug-fig2.svg)

*Figure 2: The three forms of regularization on the data points of Fig. 1. Only the feature is perturbed here, the case the equivalence with ridge regression covers, so each open square sits beside its measured day at the same label. The double arrow marks the width of the uncertainty interval they are drawn from. Enlarging the training set with them replaces the solid curve $\widehat{h}$ by the dashed one, which stays within $0.10\,^{\circ}$C of the $\widehat{h}'$ of Fig. 1. Adding the penalty term $\alpha \mathcal{R}\big\{ h \big\}$ to the training error, shown below the plot, is the second form. Shrinking the hypothesis space to $\{h : |w_{2}| \leq 0.06\}$ is the third; the hypothesis it yields lands within $0.20\,^{\circ}$C of the dashed curve and is not drawn separately. Data generated by pythondemos/dataaug.py*

Instead of adding noise to the features and labels, perturbations can also come from a transformation $\mathcal{T}$ of the feature space. A transformation known to leave the label alone is a symmetry of the learning task. A rotated or cropped image of a cat still shows a cat, while its feature vector, obtained by stacking the pixel color intensities, changes. The transformation must preserve the label, and a rotation does so only up to a point. Fig. 3 reduces images of the handwritten digits 6 and 9 to their first two principal directions (see principal component analysis) as features $x_{1}, x_{2}$. A small rotation $\mathcal{T}$ keeps an image of a 6 among the other 6s. However, rotating an image by $180$ degrees carries it into the 9s, where the label has changed.

![Figure 3 of the entry dataaug](https://dictionaryofml.org/terms/fig/dataaug-fig3.svg)

*Figure 3: Images of the handwritten digits 6 (circles) and 9 (triangles), each represented by two features $x_{1}, x_{2}$ obtained from principal component analysis. The solid line with square marks is the path that one image of a 6 traces as it is rotated from $0$ to $180$ degrees. The three inlays above the plot are that image upright, turned by $30$ degrees, and turned by $180$ degrees, each placed above its own point of the path. Its first steps keep the image among the 6s, so a rotation by a small angle preserves the label. Its last steps place the image among the 9s. Data generated by pythondemos/dataaug.py*

**See also:** [regularization](https://dictionaryofml.org/terms/regularization.html), model pruning, penalty term, ridge regression, [overfitting](https://dictionaryofml.org/terms/overfitting.html), [empirical risk minimization](https://dictionaryofml.org/terms/erm.html), [training set](https://dictionaryofml.org/terms/trainset.html), [dataset](https://dictionaryofml.org/terms/dataset.html), [hypothesis space](https://dictionaryofml.org/terms/hypospace.html), [hypothesis](https://dictionaryofml.org/terms/hypothesis.html), [data point](https://dictionaryofml.org/terms/datapoint.html), [feature](https://dictionaryofml.org/terms/feature.html), [label](https://dictionaryofml.org/terms/label.html), [feature vector](https://dictionaryofml.org/terms/featurevec.html), feature space, uncertainty, [hyperparameter](https://dictionaryofml.org/terms/hyperparameter.html), [validation error](https://dictionaryofml.org/terms/valerr.html), polynomial regression, learning task.

## References

- Bishop (1995). Training with Noise is Equivalent to Tikhonov Regularization. *Neural Computation*. <https://doi.org/10.1162/neco.1995.7.1.108>
- Goodfellow et al. (2016). Deep Learning. *MIT Press*.

## Cite this entry

```bibtex
@misc{dictml_dataaug,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {data augmentation},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/dataaug.html}
}
```