Dictionary of Applied Machine Learning

transformer

Updated on 2026-09-14

Typeset PDF version — the authoritative form of this entry

Python demo — a script that recomputes what this entry states and prints one line per check

A transformer is an artificial neural network (ANN) built by composing layers, each of which transforms the features of a data point made up of smaller units, so-called tokens, such as the words of a text or the patches of an image. The feature vectors of the tokens are stacked into a matrix; each layer maps this matrix to one of the same shape, so that layers can be composed freely into a deep ANN. The layers alternate between token mixing, which combines information across tokens, and a position-wise multilayer perceptron (MLP), which transforms each token separately.

Definition

Many machine learning (ML) applications involve data points that consist of smaller units, so-called tokens: a text consists of words, and an image consists of patches of pixels (see Fig. 1). A transformer is an artificial neural network (ANN) that is built by composing layers, each of which transforms the features of such a token-based data point (Vaswani et al., 2017).

B-riverThe feature vectors (or embeddings) $\featurevec^{(1)}, \ldots, \featurevec^{(\contextlen)} \in \reals^{\featuredim}$ of the $\contextlen$ tokens are stacked into a matrix $\mX = \big(\featurevec^{(1)}, \ldots, \featurevec^{(\contextlen)}\big)^{\top} \in \reals^{\contextlen \times \featuredim}$. Each layer of the original transformer maps such a matrix to a transformed matrix of the same shape, so that layers can be composed freely into a deep ANN (Vaswani et al., 2017, Sect. 3.1). Some transformer variants reduce the number of tokens in deeper layers (Tay et al., 2022). For example, a transformer that translates a sentence transforms the stacked word embeddings, layer by layer, into feature vectors from which the translated words are predicted (Vaswani et al., 2017). Contemporary large language models (LLMs), such as GPT-3, are transformers trained to predict the next token of a text (Brown et al., 2020); transformers also process images by taking patches as tokens (Dosovitskiy et al., 2021).

Figure 1 of the entry transformer
Figure 1: Two data points that consist of tokens. Left: the opening words of the Universal Declaration of Human Rights (Assembly, 1948); each word is one token. Right: a weather-radar-based image of the 1-hour precipitation around Krems an der Donau, Austria (August 20, 2026, 17:00 UTC; INCA analysis, GeoSphere Austria), with $96 \times 96$ pixels of 1 km; darker pixels indicate more precipitation, and thin lines cut the image into $16 \times 16$-pixel patches, each of which is one token. The course of the Danube and the location of Krems are marked (river course: OpenStreetMap contributors). Data generated by pythondemos/transformer.py
The layers of a transformer alternate between two types of transformations of $\mX$ (see Fig. 2). A token-mixing layer maps $\mX$ to a new matrix $\widetilde{\mX}$, each row of which is a weighted sum of the rows of $\mX$, with weights that can depend on $\mX$ itself; the tokens thereby interact. In the original transformer, token mixing is implemented by multi-head attention (Vaswani et al., 2017, Sect. 3.1); attention sets transformers apart from recurrent neural networks (RNNs), which process the tokens of a data point sequentially instead of in parallel (Vaswani et al., 2017, Sect. 1). A position-wise multilayer perceptron (MLP) then transforms each row of $\widetilde{\mX}$ separately (Vaswani et al., 2017, Sect. 3.3). Each of the two transformations is wrapped in a residual connection, and a subsequent layer normalization standardizes each row of the resulting matrix (Vaswani et al., 2017, Sect. 3.1).
Figure 2 of the entry transformer
Figure 2: One transformer block. Each of the two transformations, multi-head attention for token mixing and a position-wise multilayer perceptron, is bypassed by a residual connection (dashed) whose output is added back and passed through layer normalization. The largest Llama 3 LLM stacks more than 120 of such blocks (Grattafiori and others, 2024, Sect. 3.2)
The matrices produced by successive layers represent the tokens at increasingly abstract levels. This progression is measured by probing: a simple classifier is trained to predict a chosen property from the output of a single layer, and the depth at which the property becomes predictable locates it in the network. Early layers of a language transformer encode word-level properties, such as part-of-speech tags, while relations between tokens far apart in the text become predictable only in deeper layers (Tenney et al., 2019).

Deep convolutional neural networks (CNNs) show the same progression, from edges and textures in early layers to object parts in deeper ones (Zeiler and Fergus, 2014), and vision transformers develop it too: their early layers split attention between nearby and distant patches, while the deeper layers attend across the whole image (Raghu et al., 2021). The most informative representations often arise in the middle layers, whose outputs can outperform those of the final layer as features for other tasks (Skean et al., 2025).

Attention compares every pair of tokens, so its cost grows as $\contextlen^{2}$ with the number $\contextlen$ of tokens (Tay et al., 2022). This cost motivates cheaper token mixing: any transformation that combines the rows of $\mX$ can take the place of attention — a multilayer perceptron applied across tokens (Tolstikhin et al., 2021), a fixed Fourier transform (Lee-Thorp et al., 2022), or a state-space layer whose cost is linear in $\contextlen$ (Gu and Dao, 2024).

See also: attention, artificial neural network, layer, token, embedding, word embedding, recurrent neural network, natural language processing, large language model, multilayer perceptron, residual connection, layer normalization, convolutional neural network.

References

  1. Vaswani et al. (2017). Attention is all you need. Adv. Neural Inf. Process. Syst.. papers.nips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
  2. Tay et al. (2022). Efficient Transformers: A Survey. ACM Computing Surveys. doi.org/10.1145/3530811
  3. Brown et al. (2020). Language Models are Few-Shot Learners. Adv. Neural Inf. Process. Syst.. proceedings.neurips.cc/paper_files/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html
  4. Dosovitskiy et al. (2021). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. Int. Conf. Learn. Represent. (ICLR). openreview.net/forum?id=YicbFdNTTy
  5. Tenney et al. (2019). BERT Rediscovers the Classical NLP Pipeline. Proc. 57th Ann. Meeting Assoc. Comput. Linguistics. doi.org/10.18653/v1/P19-1452
  6. Zeiler and Fergus (2014). Visualizing and Understanding Convolutional Networks. Proc. European Conf. Comput. Vis. (ECCV).
  7. Raghu et al. (2021). Do Vision Transformers See Like Convolutional Neural Networks?. Adv. Neural Inf. Process. Syst..
  8. Skean et al. (2025). Layer by Layer: Uncovering Hidden Representations in Language Models. Proc. Int. Conf. Mach. Learn. (ICML).
  9. Tolstikhin et al. (2021). MLP-Mixer: An all-MLP Architecture for Vision. Adv. Neural Inf. Process. Syst.. doi.org/10.48550/arXiv.2105.01601
  10. Lee-Thorp et al. (2022). FNet: Mixing Tokens with Fourier Transforms. Proc. Conf. North Amer. Chapter Assoc. Comput. Linguistics. doi.org/10.18653/v1/2022.naacl-main.319
  11. Gu and Dao (2024). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. Conf. Language Modeling. openreview.net/forum?id=tEYskw1VY2
  12. Assembly (1948). Universal Declaration of Human Rights (UDHR). United Nations. www.un.org/en/about-us/universal-declaration-of-human-rights
  13. Grattafiori and others (2024). The Llama 3 Herd of Models. doi.org/10.48550/arXiv.2407.21783

Cite this entry

@misc{dictml_transformer,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {transformer},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-09-21},
  url = {https://dictionaryofml.org/terms/transformer.html}
}