Dictionary of Applied Machine Learning

pretraining

Updated on 2026-08-29

Typeset PDF version — the authoritative form of this entry

Pretraining is the training of a model on a large, generic dataset to produce model parameters that are reused as the starting point for a later learning task. The loss minimized is typically self-supervised, so no hand-annotated labels are needed and the dataset can be scaled up cheaply. For a large language model (LLM), pretraining minimizes the average next-token prediction loss over a large text corpus. Its output is a set of model parameters $\widehat{\weights}$, equivalently a learned feature transformation, capturing structure shared across many learning tasks. That is what a foundation model provides, and fine-tuning adapts it to one learning task. Generic features are therefore learned once rather than once per task.

Definition

Before a deep net is asked to detect tumors in a few hundred annotated scans, it is fitted to millions of ordinary photographs that have nothing to do with tumors. That first step is pretraining: the training of a model on a large, generic dataset to produce model parameters that are reused as the starting point for a later learning task.

The loss minimized during pretraining is typically self-supervised (see self-supervised learning): the supervisory signal is constructed from the input data itself, so no hand-annotated labels are needed and the dataset can be scaled up cheaply. For a large language model (LLM), pretraining minimizes the average next-token prediction loss over a large text corpus (Brown et al., 2020). The photographs in the tumor example come from ImageNet, a corpus of more than three million labeled images at its release (Deng et al., 2009).

The output is a set of model parameters $\widehat{\weights}$, or equivalently a learned feature transformation $\featuretrafo \colon \inspace \to \reals^{\featuredim}$, that captures structure shared across many learning tasks. This is what a foundation model provides: the same $\featuretrafo$ is reused across learning tasks by transfer learning, for instance by appending a small task-specific hypothesis and adapting it through fine-tuning on a task-specific dataset (see Fig. 1). Generic features are therefore learned once, and each later learning task starts from an informative initialization instead of from scratch.

Figure 1 of the entry pretraining
Figure 1: Pretraining fits model parameters $\widehat{\weights}$ to a large dataset that needs no labels, because the loss is built from the data itself. The learned feature transformation $\featuretrafo$ is then reused: each later learning task starts from it and is adapted by fine-tuning on its own small dataset, so generic features are learned once rather than once per task
Synonyms: pre-training.

See also: fine-tuning, self-supervised learning, foundation model, transfer learning, deep net.

References

  1. Brown et al. (2020). Language Models are Few-Shot Learners. Adv. Neural Inf. Process. Syst.. proceedings.neurips.cc/paper_files/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html
  2. Deng et al. (2009). ImageNet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition. doi.org/10.1109/CVPR.2009.5206848

Cite this entry

@misc{dictml_pretraining,
  author = {Jung, Alexander and Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {pretraining},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-09-19},
  url = {https://dictionaryofml.org/terms/pretraining.html}
}