Dictionary of Applied Machine Learning: 114 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

pretraining

Updated on 2026-10-29

Readiness 100/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Markdown Cite this entry

See also transfer learning fine-tuning self-supervised learning

Pretraining is the training of a model on a large, generic dataset to produce model parameters that are reused as the starting point for a later learning task. The loss minimized is typically self-supervised, so no manual annotation labour is needed. For a large language model (LLM), pretraining minimizes the average next-token prediction loss over a large text corpus. While transfer learning is the reuse of what was learned on one learning task for another, pretraining is the stage that produces what is reused, with a proxy task chosen for that purpose. The reuse takes several forms, by fine-tuning all model parameters from the pretrained ones, or by training only a subset of them on top of a pretrained feature transformation.

Figure produced by the Python demo of the entry pretraining
Figure written by the entry's Python demo pretraining.py, which recomputes what the entry states: the script block by block · open in Colab

Definition

Before a deep net is trained to detect tumors in a few hundred annotated scans, it is trained on millions of ordinary photographs that have nothing to do with tumors (Esteva et al., 2017). That first step is pretraining: the training of a model on a large, generic dataset to produce model parameters that are reused as the starting point for a later learning task.

The loss minimized during pretraining is typically self-supervised (see self-supervised learning): the label of a data point is constructed from the raw data itself, so no manual annotation labour is needed. For a large language model (LLM), pretraining minimizes the average next-token prediction loss over a large text corpus (Brown et al., 2020). The model parameters learned during pretraining are then adapted separately for each task-specific dataset by fine-tuning (Fig. 1). In the tumor example, the task-specific data points are the annotated scans, while the generic dataset is ImageNet, a corpus of more than three million labeled images at its release (Deng et al., 2009), so that pretraining is supervised, the exception to the self-supervised rule.

B-pretrainThe reuse takes a second form besides fine-tuning: a pretrained feature transformation $\phi \colon \mathcal{X} \to \mathbb{R}^{d}$ is concatenated with a task-specific model. In a deep net, the first layers are the pretrained $\phi$ and the final layers are trained separately for each learning task (Fig. 2) (Yosinski et al., 2014). The same $\phi$ serves every learning task, so generic features are learned once, and each later learning task starts from an informative initialization instead of from scratch (Yosinski et al., 2014).

Figure 1 of the entry pretraining
Figure 1: Pretraining learns model parameters $\widehat{\weights}$ from a large dataset with labels constructed from the raw data. Each later learning task fine-tunes $\widehat{\weights}$ on its own small dataset
Figure 2 of the entry pretraining
Figure 2: A deep net whose first two layers are the pretrained feature transformation $\featuretrafo$ (dashed box). Pretraining learns these layers once on the large generic dataset. The output layer is trained separately on the small dataset of each learning task, on top of the same $\featuretrafo$
A worked example. A piece of an aerial photograph of the Wachau, $2.4$ km by $1.2$ km, is cut into $4608$ patches of $25$ m. A parcel register labels a patch as vineyard when more than half of its pixels lie on vineyard parcels, which is the case for a third of them. The western half is the training set, the eastern half the test set. At this resolution the rows of vines are visible stripes, while the mean color of a patch tells little: vineyards, meadows and gardens are all green (Fig. 3).
Figure 3 of the entry pretraining
Figure 3: The piece of the aerial photograph of the Wachau (left) and the parcel register that labels its patches (right): a patch counts as vineyard when more than half of its pixels lie on a vineyard parcel, shown white. The dashed line is the split into the western training set and the eastern test set. Photograph: basemap.at (CC BY 4.0); parcels: INVEKOS 2025, AMA (CC BY 3.0 AT). Data generated by pythondemos/pretraining.py
Pretraining uses the photograph alone, never the labels. The central $8 \times 8$ pixel block of a western patch is blanked. A convolutional neural network (CNN) of two convolutional layers, each with eight $3 \times 3$ filters and a rectified linear unit (ReLU), is followed by a linear readout at the blanked block. It is trained to predict the mean color of the blanked pixels from their surroundings. On the eastern patches the trained network predicts the blanked color with a squared error of $0.0008$, a quarter of the error of the mean color of the rest of the patch (Fig. 4, top).

B-rotationA second pretext task trains the same network to tell whether a patch was rotated by $0$, $90$, $180$ or $270$ degrees. The direction of the rows and of the shadows answers that question. The network recognizes the rotation of $37$ percent of the eastern patches, against $25$ by chance.

The two convolutional layers are then frozen: they are the feature transformation $\phi$, and their output on the original patch, averaged over a $4 \times 4$ grid, gives $128$ features. Logistic regression on these features is a linear classifier of vineyard against no vineyard. It is trained on as few as $8$ and at most $128$ labeled data points of the western half, and its accuracy is measured on the eastern half (Fig. 4, bottom). Labels are scarce on purpose: this is the regime in which pretraining decides the outcome.

With so few labels a linear classifier on the pixels themselves is prone to overfitting. It has $768$ features, the patch pooled to $16 \times 16$ pixels in three colors, and at most $128$ patches to learn them from, and its accuracy stays below the majority baseline of $0.69$ until $128$ labels are available, where it reaches $0.72$. The mean color of a patch stays at that baseline too, $0.67$ with $128$ labels. The control that every pretraining claim needs is the same two layers with their random initial weights, never trained: they reach $0.76$ with $128$ labels, since random convolutional filters with pooling already respond to texture (Saxe et al., 2011). The color-pretrained layers exceed that control from $32$ labels on, and the rotation-pretrained layers exceed it at every number of labels, reaching $0.82$ with $128$.

B-transferTwo conclusions follow. Pretraining helps when the labeled dataset is too small to learn what the pretext task already taught the layers. The pretext task matters too. A task that needs the direction of the stripes yields features that a classifier of vineyards needs, while predicting a color from its surroundings is largely solved by averaging. The numbers are computed by pythondemos/pretraining.py.

Figure 4 of the entry pretraining
Figure 4: Top: eight eastern patches of $25$ m with the blanked center (white), the center color predicted by the pretrained CNN, and the original. Bottom: accuracy of a linear classifier of vineyard patches on the eastern half against the number of labeled western patches, for five feature sets: the mean color (dotted gray curve, filled triangles), the pooled patch (dashed gray curve, filled squares), untrained layers (solid gray curve, filled circles), layers pretrained to predict the blanked color (solid black curve, filled diamonds), layers pretrained to recognize the rotation (solid black curve, inverted filled triangles), and the majority baseline (dotted black line). Photograph: basemap.at (CC BY 4.0); parcels: INVEKOS 2025, AMA (CC BY 3.0 AT). Data generated by pythondemos/pretraining.py

Synonyms: pre-training.

See also: transfer learning, fine-tuning, self-supervised learning, foundation model, multitask learning, deep net.

References

  1. Esteva et al. (2017). Dermatologist-level classification of skin cancer with deep neural networks. Nature.
  2. Brown et al. (2020). Language Models are Few-Shot Learners. Adv. Neural Inf. Process. Syst.. proceedings.neurips.cc/paper_files/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html
  3. Deng et al. (2009). ImageNet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition. doi.org/10.1109/CVPR.2009.5206848
  4. Yosinski et al. (2014). How Transferable Are Features in Deep Neural Networks?. Adv. Neural Inf. Process. Syst. (NIPS).
  5. Saxe et al. (2011). On Random Weights and Unsupervised Feature Learning. Proceedings of the 28th International Conference on Machine Learning (ICML).

Cite this entry

@misc{dictml_pretraining,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {pretraining},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-03},
  url = {https://dictionaryofml.org/terms/pretraining.html}
}