Python demo — a script that recomputes what this entry states and prints one line per check
Fine-tuning continues the training of a model on a
task-specific dataset, starting from model parameters obtained
by pretraining rather than from a fresh initialization. It solves
an empirical risk minimization (ERM) problem over a small training set, with the gradient descent (GD)
iteration warm-started at the pretrained model parameters. A few
GD steps move them a bounded distance, so the hypotheses
the method can return lie in a neighborhood of the pretrained one
rather than anywhere in the hypothesis space. That is a restriction of
the model the method searches, imposed by the initialization
instead of by a penalty term, and it is why a small training set
suffices. Variants freeze some layers, or restrict the update to
a low-dimensional set of new model parameters.
Definition
B-tasksA hospital has a few hundred annotated
scans and needs a tumor detector. Training a deep net from a
fresh initialization on that many labeled data points usually
ends in overfitting; starting from model parameters already
fitted to millions of unrelated photographs works markedly better
(Yosinski et al., 2014). Fine-tuning is the second step of that
recipe: it continues the training of a model on a
task-specific dataset, starting from model parameters
$\widehat{\weights}^{(\mathrm{pre})}$ obtained by pretraining
rather than from a fresh initialization. Large language models (LLMs) are adapted the
same way: one pretrained on unlabeled text is fine-tuned on each
task it is wanted for (Devlin et al., 2019).
Formally, fine-tuning solves the empirical risk minimization (ERM) problem over a
training set
$\trainset = \{\datapoint^{(\sampleidx)}\}_{\sampleidx=1}^{\samplesize}$,
\[
\widehat{\weights}
\in \argmin_{\weights}
\frac{1}{\samplesize}
\sum_{\sampleidx=1}^{\samplesize}
\lossfunc{\datapoint^{(\sampleidx)}}{\weights}
\text{,}
\]
but the gradient descent (GD) iteration is warm-started at the pretrained
model parameters, $\weights^{(0)} =
\widehat{\weights}^{(\mathrm{pre})}$, instead of a fresh
initialization.
B-warmThe warm start is what makes a small training set enough. A few
GD steps with step size $\lrate$ move the
model parameters a bounded distance from where they started, so the
hypotheses the method can actually return lie in a
neighborhood of the pretrained one rather than anywhere in the
hypothesis space (see Fig. 2). That restricts the model the
method searches, which is what regularization achieves by pruning
a hypothesis space, except that here the restriction comes from the
initialization and no term is added to the objective function. The
restriction is what a small training set needs in the probabilistic
formalization: for data points that are realizations of
independent and identically distributed (i.i.d.) random variables (RVs), a smaller set of reachable hypotheses
narrows the generalization gap between training error and risk —
for a finite hypothesis space, to at most
$\sqrt{\log(2 |\hypospace| / \delta) / (2 \samplesize)}$ with
probability at least $1 - \delta$
(Shalev-Shwartz and Ben-David, 2014, Cor. 4.6). The experiment
pythondemos/finetuning.py measures the effect for
linear regression with $\samplesize = 15$ data points and
$\featuredim = 30$ model parameters: warm-started GD dips
to a validation error below twice the noise level within twenty steps,
moving the model parameters a distance $0.6$, while the same
iteration from a fresh initialization drives the training error to
zero yet ends with a validation error more than four times larger,
having moved a distance $2.6$. A common variant freezes some
layers and adapts only the remainder (Yosinski et al., 2014);
parameter-efficient fine-tuning restricts the update to a
low-dimensional set of new model parameters while leaving
$\widehat{\weights}^{(\mathrm{pre})}$ untouched
(Hu et al., 2022).
Fig. 1 shows the effect in the
feature–label plane.
Figure 1: Fine-tuning in the feature–label plane. The pretrained
hypothesis (dashed) is already close to the task; a few
GD steps on the four data points of the task
training set (dots) nudge it to the solid curve. From a fresh
initialization, the four data points alone would have to
determine the whole curveFigure 2: Fine-tuning in the space of model parameters. A few
GD steps on a small training set cannot carry the
model parameters far, so the hypotheses the method can
return lie in the dashed neighborhood of
$\widehat{\weights}^{(\mathrm{pre})}$ rather than anywhere in the
hypothesis space. Reaching the same region from a fresh
initialization (dotted) would take far more labeled data points
Synonyms: fine tuning.
Yosinski et al. (2014). How Transferable Are Features in Deep Neural Networks?. Adv. Neural Inf. Process. Syst. (NIPS).
Devlin et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proc. 2019 Conf. North American Chapter Assoc. Computational Linguistics: Human Lang. Technologies, Volume 1 (Long and Short Papers). doi.org/10.18653/v1/N19-1423
Shalev-Shwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. Cambridge Univ. Press. doi.org/10.1017/cbo9781107298019
Hu et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. Int. Conf. Learn. Represent. (ICLR).
Cite this entry
@misc{dictml_finetuning,
author = {Jung, Alexander and Olioumtsevits, Konstantina and Schnoor, Ekkehard},
title = {fine-tuning},
howpublished = {Dictionary of Applied Machine Learning (course edition)},
year = {2026},
doi = {10.5281/zenodo.21569296},
note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-09-19},
url = {https://dictionaryofml.org/terms/finetuning.html}
}
