finetuning.py, which recomputes what the entry states: the script block by block · open in ColabDictionary of Applied Machine Learning: 114 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here
Updated on 2026-10-29
Readiness 100/100
See also pretraining transfer learning self-supervised learning empirical risk minimization gradient descent regularization overfitting validation error
Fine-tuning continues the training of a model on a task-specific dataset, starting from model parameters obtained by pretraining on a larger dataset. It solves an empirical risk minimization (ERM) problem over a small training set under the constraint that the model parameters stay close to the pretrained ones. One form of fine-tuning runs a few gradient descent (GD) steps on the small training set, started at the pretrained model parameters. Forcing the model parameters to stay close to the pretrained ones is a form of regularization, since it limits the effective hypothesis space that is searched during fine-tuning. For a linear model, the generalization gap grows in proportion to how far the model parameters may move from the pretrained ones, which is why a small training set suffices.
finetuning.py, which recomputes what the entry states: the script block by block · open in ColabB-warmConsider daily weather at Krems an der
Donau. A linear model that predicts the maximum daytime temperature
from the morning minimum is trained on 360 days of one year, which
yields a pretrained hypothesis. The
task-specific datasets are the temperature measurements of a
single month, and one task is to learn a forecaster tailored to that
month. For a single month, thirty gradient descent (GD) steps on that month's 30
days adjust the model parameters, starting from those learned on
the whole year (Fig. 1). Fine-tuning
is this second step: it continues the training of a
model on a task-specific dataset, starting from
model parameters obtained by pretraining on a larger
dataset (Bishop and Bishop, 2024, Sect. 6.3.4).
pythondemos/finetuning.py
The generalization gap between risk and training error $f({\mathbf w})$ therefore grows in proportion to the radius $B$ and not with the number $d$ of model parameters. A gap of at most $\varepsilon$ is guaranteed by $m = \big\lceil \big(2 \rho B R + c \sqrt{2 \ln(2/\delta)}\big)^{2} / \varepsilon^{2} \big\rceil$ data points, a number that falls with $B$. The constraint restricts the model the method searches, which is what regularization achieves by pruning a hypothesis space. The difference is that the pruned set is centered at the pretrained model parameters instead of at an arbitrary initialization.
B-distanceGD with few steps and a small step size, started at the pretrained model parameters, implements this constraint without enforcing it explicitly. Each step moves the model parameters by $\eta$ times the norm of the gradient. After $T$ steps with step size $\eta$, started at ${\mathbf w}^{(0)} = \widehat{{\mathbf w}}^{(\mathrm{pre})}$, \[ \Vert {{\mathbf w}^{(T)} - \widehat{{\mathbf w}}^{(\mathrm{pre})}} \Vert \leq \eta \sum_{t=0}^{T-1} \Vert {\nabla f({\mathbf w}^{(t)})} \Vert \leq \eta T G \text{,} \] where $G$ bounds the norm of the gradient along the iterates. The iterate therefore lies in the ball of radius $B = \eta T G$ around $\widehat{{\mathbf w}}^{(\mathrm{pre})}$. Limiting the number of iterations is thus a form of regularization: it limits the effective hypothesis space that is searched during fine-tuning, with the step size and the iteration count in the role of the radius.
Fine-tuning is useful when the task-specific dataset is small. A hospital has a few hundred annotated scans and needs a tumor detector. Training a deep net from an arbitrary initialization on that many labeled data points usually ends in overfitting; starting from model parameters already learned from millions of unrelated photographs achieves a lower validation error (Yosinski et al., 2014). Large language models (LLMs) are adapted the same way: one pretrained on unlabeled text is fine-tuned on each task it is wanted for (Devlin et al., 2019). A common variant freezes some layers and adapts only the remainder (Yosinski et al., 2014); parameter-efficient fine-tuning restricts the update to a low-dimensional set of new model parameters while leaving $\widehat{{\mathbf w}}^{(\mathrm{pre})}$ untouched (Hu et al., 2022).
The experiment pythondemos/finetuning.py measures the
effect on synthetic data for linear regression with $m = 15$
data points
and $d = 30$ model parameters. GD started at the
pretrained model parameters with step size $\eta = 0.02$
falls to a validation error below twice the noise level after $T =
16$ steps. It has then moved the model parameters a distance
$0.32$, inside the ball of radius $\eta T G = 0.72$. The
same iteration from an arbitrary initialization drives the
training error to zero yet ends with a validation error more than four
times larger, having moved a distance $2.6$.
Fig. 3 sketches the same effect on a
schematic task-specific dataset of four data points.
Synonyms: fine tuning.
See also: pretraining, transfer learning, foundation model, self-supervised learning, empirical risk minimization, gradient descent, regularization, Rademacher complexity, overfitting, generalization gap, validation error.
@misc{dictml_finetuning,
author = {Jung, Alexander},
editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
title = {fine-tuning},
howpublished = {Dictionary of Applied Machine Learning (course edition)},
year = {2026},
doi = {10.5281/zenodo.21569296},
note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-03},
url = {https://dictionaryofml.org/terms/finetuning.html}
}