pythondemos/supervisedlearning.pyDictionary of Applied Machine Learning
Updated on 2026-09-20
Typeset PDF version — the authoritative form of this entry
Python demo — a script that recomputes what this entry states and prints one line per check
Supervised learning is the machine learning (ML) setting in which every data point of the training set carries a label, the quantity to be predicted, alongside its features. The goal is to learn a hypothesis that maps the feature vector of a data point to its label and incurs a small loss on data points outside the training set. The standard formalization is empirical risk minimization (ERM), which chooses a hypothesis minimizing the average loss on the training set; without labels that average cannot be formed. The label space decides the learning task: a numeric label makes it regression, a discrete one classification. Labels do not by themselves deliver generalization, since a hypothesis space large enough to fit the training set exactly can incur an arbitrarily large loss elsewhere. Because labels are often expensive to obtain, unsupervised learning, self-supervised learning and semi-supervised learning (SSL) each reduce what must be annotated.
A weather station at Krems records, for every day of 2024, the minimum temperature in the morning and the maximum temperature during the day. To forecast the afternoon from the morning, each day is taken as a data point: its feature is the morning minimum, and the maximum of that day is the quantity to be predicted. For the days already recorded, that quantity is known, because it was measured. Supervised learning is the machine learning (ML) setting in which every data point of the training set comes with the quantity to be predicted, its label (Sutton and Barto, 2018, Sect. 1.1; Hastie et al., 2009, Ch. 2).
Formally, the training set consists of labeled data points $\trainset = \big\{ \pair{\featurevec^{(\sampleidx)}}{\truelabel^{(\sampleidx)}} \big\}_{\sampleidx=1}^{\samplesize}$ with feature vectors $\featurevec^{(\sampleidx)} \in \featurespace$ and labels $\truelabel^{(\sampleidx)} \in \labelspace$. The goal is to learn a hypothesis $\hypothesis: \featurespace \rightarrow \labelspace$ that incurs a small loss on data points outside $\trainset$, which is what generalization demands. That requirement is made precise by the independent and identically distributed assumption (i.i.d. assumption): the data points are realizations of independent and identically distributed (i.i.d.) random variables (RVs) with a common probability distribution, and the quantity to be small is the risk, the expected loss under it. The labels make this a learning task that can be formalized as empirical risk minimization (ERM): among the hypotheses of a hypothesis space $\hypospace$, choose one that minimizes the average loss on the training set, \[ \learnthypothesis \in \argmin_{\hypothesis \in \hypospace} \frac{1}{\samplesize} \sum_{\sampleidx=1}^{\samplesize} \lossfunc{\pair{\featurevec^{(\sampleidx)}}{\truelabel^{(\sampleidx)}}}{\hypothesis} \text{.} \] Without labels this average cannot be formed, since the loss of a prediction is defined only against the label it is compared with. Whether the learned hypothesis is any good is measured on data points kept out of training, the validation set. Fig. 1 shows both for the days at Krems, with a hypothesis delivered by ERM over the linear model.
B-classifyThe label space decides which learning task it is. A
numeric label space $\labelspace = \reals$ makes the task
regression, a discrete one makes it classification. The weather records deliver both from the same data points. With
the maximum temperature of the day as label the task is
regression. With the label frost in the morning,
which is positive on $19\%$ of the days, it is
classification, and ERM with the $0/1$ loss, which counts the
data points whose label is predicted wrongly, delivers
a classifier that is correct on $81\%$ of the
held-out days, against $71\%$ for always answering with the more
frequent of the two labels.
pythondemos/supervisedlearning.py
The labels that make all of this possible are not free. Where one comes from varies: it can be measured, as the maximum temperature of a day is; it can be recorded, as the price at which a house sold is; or it must be annotated by a person, as the diagnosis shown by a medical image is. In the last case the labels are the scarce resource: annotating requires the expert whose judgment is to be reproduced, so a dataset of unlabeled data points is typically much cheaper to enlarge than a training set of labeled data points. Assembling ImageNet, a dataset of more than three million annotated images, took tens of thousands of people working through a crowdsourcing platform (Deng et al., 2009). Three settings answer this shortage differently. Unsupervised learning drops the labels entirely and looks for structure in the feature vectors alone. Self-supervised learning constructs the labels from the data points themselves, by withholding some of their features, and so returns an ordinary supervised problem whose training set no annotator has touched. Semi-supervised learning (SSL) uses a few labeled data points together with many unlabeled ones. Reinforcement learning (RL) has no label at all: it observes a reward for the action it took, and never the action that would have been correct (Sutton and Barto, 2018, Sect. 1.1).
See also: label, labeled data point, training set, validation set, classifier, independent and identically distributed assumption, risk, empirical risk minimization, hypothesis, classification, regression, generalization, overfitting, unsupervised learning, self-supervised learning, semi-supervised learning, reinforcement learning.
@misc{dictml_supervisedlearning,
author = {Jung, Alexander and Olioumtsevits, Konstantina and Schnoor, Ekkehard},
title = {supervised learning},
howpublished = {Dictionary of Applied Machine Learning (course edition)},
year = {2026},
doi = {10.5281/zenodo.21569296},
note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-09-21},
url = {https://dictionaryofml.org/terms/supervisedlearning.html}
}