Dictionary of Applied Machine Learning: 114 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

transfer learning

Updated on 2026-10-29

Readiness 100/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Markdown Cite this entry

See also fine-tuning

Transfer learning is the reuse of what was learned on one learning task to solve another learning task. A learning task can be characterized by a probability distribution of data points, and the effectiveness of transfer learning depends on the similarity of the corresponding probability distributions. Task similarity can be encoded as a similarity graph whose edge weights couple the learning tasks at its nodes; transfer learning methods can then be obtained by optimization over this graph.

Figure produced by the Python demo of the entry transfer learning
Figure written by the entry's Python demo transferlearning.py, which recomputes what the entry states: the script block by block · open in Colab

Definition

B-twoThe weather stations Krems an der Donau and Melk in the Wachau valley each record the minimum and the maximum temperature of every day. Predicting the maximum temperature $y$ of a day at one station, the label, from its minimum temperature $x$, the feature, is a learning task. Each weather station defines a separate learning task. Fig. 1 shows the two tasks side by side, each with a decision tree of depth two, the hypothesis learned from a training set of five labeled days of 2020. A learning task can be characterized by the probability distribution of its data points: at the two stations those distributions are similar. Transfer learning uses such similarity. It couples the learning tasks so that what is learned at one station improves the predictions at the other, and its effectiveness depends on how similar the probability distributions of the coupled tasks are (Pan and Yang, 2010).

Figure 1 of the entry transferlearning
Figure 1: Two learning tasks in the Wachau valley. (a) The course of the Danube and four weather stations around the valley. (b) and (c) The days of 2020 at Krems an der Donau and at Melk, with the minimum temperature of a day as feature $\feature$ and its maximum temperature as label $\truelabel$. At each station, five labeled days (filled circles) are the training set. The dotted step function is the decision tree learned from those five days alone; the solid one is the decision tree learned when the two tasks are coupled by requiring their predictions to agree on the test features $\mathcal{T}$ ($20$ of the $40$, every second, as ticks along the horizontal axis). Data generated by pythondemos/transferlearning.py
The coupling in Fig. 1 is implemented by adding a penalty to the objective function of the model training at each station. Let $h^{(1)}$ and $h^{(2)}$ be the hypotheses of the two stations and $\mathcal{T}$ a test set of feature values selected at random from the range $[-10, 10]$ of minimum temperatures. Transfer learning then minimizes the sum of the two training errors plus $\alpha \frac{1}{|\mathcal{T}|} \sum_{x \in \mathcal{T}} \big( h^{(1)}(x) - h^{(2)}(x) \big)^{2}$, the mean squared disagreement of the two hypotheses on the test features. For decision trees, minimizing over one tree while the other is fixed is a weighted fit of that tree to its own days together with the test features labeled by the other tree, and alternating the two fits solves the coupled problem. With five labeled days per station, the coupled trees agree on the test features (mean squared disagreement $32.2 \to 0.1$), and their average error on the held-out days of the two stations falls from $47.4$ to $33.5$.

B-graphThe coupling of two tasks extends to a collection of learning tasks, one per node $i = 1, \ldots, n$ of an undirected weighted graph $\mathcal{G}$, the similarity graph. A weighted edge $\{i,i'\}$ with edge weight $A_{i,i'}$ encodes the similarity between two learning tasks; a zero weight leaves them uncoupled. Each node $i$ carries a local model, a local dataset, and the local hypothesis $h^{(i)}$ learned from them. Minimizing the sum of the local training errors plus $\alpha$ times the generalized total variation (GTV) \[ \sum_{\{i,i'\} \in \mathcal{E}} A_{i,i'} d^{(i,i')} \text{,} \] with $d^{(i,i')}$ a measure of the discrepancy between the local hypotheses $h^{(i)}$ and $h^{(i')}$, is generalized total variation minimization (GTVMin) (SarcheshmehPour et al., 2023). The coupling in Fig. 1 is GTVMin on a graph with two nodes, with the mean squared disagreement on the test features $\mathcal{T}$ as the discrepancy measure; this measure applies to any local model, the decision trees included. For local models with model parameters $\mathbf{w}^{(i)}$, the squared distance $\left\lVert \mathbf{w}^{(i)} - \mathbf{w}^{(i')} \right\rVert_{2}^{2}$ is a common choice, and the network Lasso is the variant with the unsquared distance (Hallac et al., 2015). GTVMin learns all local hypotheses simultaneously and transfers knowledge between well-connected learning tasks, as in federated learning (FL). Fig. 2 shows a similarity graph of nine stations in Lower Austria, with edge weights that fall off with the distance between stations and are kept for the nearest neighbors of each. With five labeled days and a decision tree at every station, GTVMin with the disagreement on $\mathcal{T}$ lowers the average held-out error of the nine stations from $57.0$ to $35.9$ and the error at eight of them; at the ninth, Wien, whose edges are the weakest of the graph, the coupling raises it slightly.

Figure 2 of the entry transferlearning
Figure 2: A similarity graph of nine weather stations in Lower Austria, each a node carrying its own learning task. The edge weight between two stations is $\exp(-d^{2}/(2 s^{2}))$ for their distance $d$ and $s = 40$ kilometers, kept for the three nearest neighbors of every station: thick edges weigh at least $0.5$, thin edges less. Data generated by pythondemos/transferlearning.py
A special case of transfer learning is a node without any labeled data points. Its own training error is absent from the objective, so GTVMin determines its hypothesis through the GTV alone, from the predictions of its neighbors on the test features. This is semi-supervised learning (SSL): the unlabeled node still has data points, and its position in the similarity graph is what lets them be used (Anon., 2006).

A second form of transfer learning places the similarity between a pretraining task and the new one, instead of between the stations of a similarity graph. It adapts a deep net trained on a large dataset to a small one: the features computed by its early layers are kept (see feature learning) and the remaining model parameters are adapted by fine-tuning. When the features useful for the first task are not useful for the second, the adapted hypothesis can have a larger risk than one trained from scratch, which is called negative transfer (Pan and Yang, 2010). The case in which the learning task stays fixed and only the probability distribution of the feature vectors changes is domain adaptation (Pan and Yang, 2010, Sect. 2.3).

See also: learning task, generalized total variation minimization, generalized total variation, local model, local dataset, edge weight, semi-supervised learning, multitask learning, fine-tuning, domain adaptation, federated learning.

References

  1. Pan and Yang (2010). A Survey on Transfer Learning. IEEE Transactions on Knowledge and Data Engineering. doi.org/10.1109/TKDE.2009.191
  2. SarcheshmehPour et al. (2023). Clustered Federated Learning via Generalized Total Variation Minimization. IEEE Trans. Signal Process.. doi.org/10.1109/TSP.2023.3322848
  3. Hallac et al. (2015). Network Lasso: Clustering and Optimization in Large Graphs. Proc. 21st ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining. doi.org/10.1145/2783258.2783313
  4. Anon. (2006). Semi-Supervised Learning. MIT Press. doi.org/10.7551/mitpress/9780262033589.001.0001

Cite this entry

@misc{dictml_transferlearning,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {transfer learning},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-03},
  url = {https://dictionaryofml.org/terms/transferlearning.html}
}