# self-supervised learning *Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/selfsupervisedlearning.html) · [typeset PDF](https://dictionaryofml.org/terms/selfsupervisedlearning.pdf) · [LaTeX](https://dictionaryofml.org/terms/selfsupervisedlearning.tex) · [Python demo](https://dictionaryofml.org/terms/selfsupervisedlearning.py) · [notebook](https://dictionaryofml.org/terms/selfsupervisedlearning.ipynb)* Math is LaTeX with the book's macros expanded, so no notation package is needed to read it. ## Abstract Self-supervised learning constructs the labels of a training set from the data points themselves. Some features of a data point are withheld and used as its label while the rest remain features, so what results is an ordinary supervised learning problem, solved by empirical risk minimization on data that no annotator has touched. In natural language processing the withheld feature is the next token, which is how a large language model is trained. In computer vision it is a set of pixels, deleted from an image and predicted from the patches left visible. ## Definition An open image database returns thousands of photographs for a keyword such as *cat*, and a web crawl yields billions of sentences. Neither comes with a label saying what should be predicted. A prediction task can be made from any one of them: hide the last word of a sentence, and the words before it become a question whose answer is already known. Self-supervised learning constructs labels this way, from the data point itself. In general, some of the features of a data point are withheld and used as the label; the remaining ones stay as the features. The result is an ordinary supervised learning problem, solved by empirical risk minimization, and its training set needs nothing beyond the raw data. A single unlabeled collection yields as many training sets as there are ways of hiding part of a data point (see Fig. 1). The term goes back to de Sa (1993), who uses it for a classifier whose label for one sensory modality is derived from a co-occurring input in another. The withheld feature can be a token. In natural language processing a data point is a stretch of text, its features are the $n$ tokens $v^{(1)}, \ldots, v^{(n)}$ of a context window of length $n$, and its label is the token $v^{(n+1)}$ that follows. Every position in every document is one labeled data point, so a corpus of $m$ tokens gives on the order of $m$ of them. This is the training task of a large language model such as GPT-3, an autoregressive language model, i.e., one trained to predict the next token (Brown et al., 2020, Sect. 1). Masking a token in the middle and predicting it from both sides withholds a different feature of the same data point (Devlin et al., 2019). Similarly, in computer vision the withheld features are pixels: an image is split into patches, most of them are deleted, and the method predicts the missing pixel values from those that remain. He et al. (2022, Sect. 1 and Fig. 2) delete $75\%$ of the patches and still recover recognizable content. A deep net solving either task computes a composition $\widehat{h} = s \circ \phi$. The feature transformation $\phi \colon \mathcal{X} \to \mathbb{R}^{d}$, with $\mathcal{X}$ the set of raw inputs (texts or images), sends a data point to a vector of activations, and the map $s$ predicts the withheld feature from that vector. That feature can be predicted only from what $\phi$ has kept: a next token cannot be recovered from a vector that has discarded the context, nor a deleted patch from one that has dropped its surroundings. Training the composition therefore constrains $\phi$ to keep that information, even though $\phi$ appears nowhere in the loss. The constraint does not determine $\phi$ uniquely: for any invertible map $g$ of $\mathbb{R}^{d}$ onto itself, the composition $(s \circ g^{-1}) \circ (g \circ \phi)$ equals $s \circ \phi$ and so has the same loss. He et al. (2022, Sect. 3) name the two halves: their encoder is $\phi$ and receives only the patches left visible, while their decoder is $s$ and rebuilds the pixels from its output. What restricts $\phi$ there is the deletion of most of the input, not a narrow layer as in an autoencoder. In both domains it is $\phi$ that is carried over. The map $s$ is dropped, and a small replacement for it is fitted on the output of $\phi$ using the few labels the later task does have (see transfer learning, foundation model). Self-supervised learning is therefore one answer to the same shortage that motivates semi-supervised learning: there, unlabeled data points assist a label-scarce problem directly; here, they are turned into a labeled problem of their own first. Unsupervised learning is the third answer, and it gives up the label entirely: it finds structure among the feature vectors instead of predicting a withheld one. ![Figure 1 of the entry selfsupervisedlearning](https://dictionaryofml.org/terms/fig/selfsupervisedlearning-fig1.svg) *Figure 1: Two ways of constructing labels from unlabeled data points, and in both a single source yields many. (a) Natural language processing: a sentence is read through a sliding window, and every position gives a labeled data point whose features are the tokens in the window and whose label is the token after it (shaded). (b) Computer vision: an image is split into patches and most are deleted (shaded); each missing patch, three of them outlined here, is one labeled data point whose label is its own pixel values. Neither task needs a human annotator* **See also:** [feature](https://dictionaryofml.org/terms/feature.html), [label](https://dictionaryofml.org/terms/label.html), [supervised learning](https://dictionaryofml.org/terms/supervisedlearning.html), [unsupervised learning](https://dictionaryofml.org/terms/unsupervisedlearning.html), semi-supervised learning, [large language model](https://dictionaryofml.org/terms/llm.html), natural language processing, token, [transfer learning](https://dictionaryofml.org/terms/transferlearning.html), foundation model. ## References - Brown et al. (2020). Language Models are Few-Shot Learners. *Adv. Neural Inf. Process. Syst.*. - Devlin et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. *Proc. 2019 Conf. North American Chapter Assoc. Computational Linguistics: Human Lang. Technologies, Volume 1 (Long and Short Papers)*. - de Sa (1993). Learning Classification with Unlabeled Data. *Adv. Neural Inf. Process. Syst. (NIPS)*. - He et al. (2022). Masked Autoencoders Are Scalable Vision Learners. *2022 IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR)*. ## Cite this entry ```bibtex @misc{dictml_selfsupervisedlearning, author = {Jung, Alexander}, editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard}, title = {self-supervised learning}, howpublished = {Dictionary of Applied Machine Learning (course edition)}, year = {2026}, doi = {10.5281/zenodo.21569296}, note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11}, url = {https://dictionaryofml.org/terms/selfsupervisedlearning.html} } ```