\documentclass{article} \usepackage[utf8]{inputenc} \usepackage{amsmath,amssymb} \usepackage{algorithm,algpseudocode} \usepackage{bm} \usepackage{hyperref} \usepackage{float} \usepackage{subcaption} \usepackage{enumitem} \usepackage{xcolor} \definecolor{darkgreen}{rgb}{0.0, 0.5, 0.0} \definecolor{aaltoBlue}{RGB}{0, 51, 102} \definecolor{pinegreen}{cmyk}{0.92,0,0.59,0.25} \definecolor{royalblue}{cmyk}{1,0.50,0,0} \definecolor{lavander}{cmyk}{0,0.48,0,0} \definecolor{violet}{cmyk}{0.79,0.88,0,0} \definecolor{lightblue}{RGB}{173, 216, 230} \usepackage{tikz} \usepackage{pgfplots} \usetikzlibrary{calc,arrows.meta,positioning,fit,shapes.geometric,shapes.misc,intersections,patterns,decorations.pathreplacing,backgrounds,matrix} \usepgfplotslibrary{fillbetween,groupplots} \setcounter{secnumdepth}{-\maxdimen} % Long unbreakable tokens (the ISO/IEC author names in citations, term names % in the title) otherwise overflow the margin by a few points; a modest % emergency stretch lets TeX loosen the paragraph instead. \emergencystretch=2em \usepackage{mleftright} % Project macros used by this entry (auto-extracted). \newcommand{\contextlen}{n} \newcommand{\featuredim}{d} \newcommand{\featurevec}{\vx} \newcommand{\mX}[0]{{\bf X}} \newcommand{\reals}{\mathbb{R}} \newcommand{\vx}[0]{{\bf x}} \begin{document} \section{transformer} \subsection*{Author and editors} Alexander Jung\textsuperscript{*}, Department of Computer Science, Aalto University, Espoo, Finland (ORCID \mbox{\href{https://orcid.org/0000-0001-7538-0990}{0000-0001-7538-0990}}) \\ Editors: Konstantina Olioumtsevits (ORCID \mbox{\href{https://orcid.org/0000-0003-2697-8659}{0000-0003-2697-8659}}), Ekkehard Schnoor (ORCID \mbox{\href{https://orcid.org/0000-0001-8252-1499}{0000-0001-8252-1499}}), Aalto University, Espoo, Finland \\ \textsuperscript{*}Corresponding author: \href{mailto:alex.jung@aalto.fi}{alex.jung@aalto.fi} \subsection{Abstract} A transformer is an artificial neural network (ANN) built by composing layers, each of which transforms the features of a data point made up of smaller units, so-called tokens, such as the words of a text or the patches of an image. The feature vectors of the tokens are stacked into a matrix; each layer maps this matrix to one of the same shape, so that layers can be composed freely into a deep ANN. The layers alternate between token mixing, which combines information across tokens, and a position-wise multilayer perceptron (MLP), which transforms each token separately. \subsection{Definition} A text consists of words, and an image consists of patches of pixels (see Fig.~\ref{fig_transformer_tokens_dict}): the data points of many machine learning (ML) applications consist of smaller units, so-called tokens. A transformer is an artificial neural network (ANN) that is built by composing layers, each of which transforms the features of such a token-based data point (Vaswani et al., 2017). \begin{figure}[H] \centering \begin{tikzpicture}[ every node/.style={font=\footnotesize}, word/.style={draw, rounded corners=1pt, inner sep=2.5pt, minimum height=5mm}, ] % left: a sentence cut into word tokens \node at (2.3,3.0) {text data point}; \node[word] (w1) at (0,1.7) {All}; \node[word, right=1.2mm of w1] (w2) {human}; \node[word, right=1.2mm of w2] (w3) {beings}; \node[word, right=1.2mm of w3] (w4) {are}; \node[word] (w5) at (0,0.9) {born}; \node[word, right=1.2mm of w5] (w6) {free}; \node[word, right=1.2mm of w6] (w7) {and}; \node[word, right=1.2mm of w7] (w8) {equal}; \node[word, right=1.2mm of w8] (w9) {$\ldots$}; \node at (2.3,-0.4) {tokens = words}; \node at (2.3,-1.0) {(a)}; % right: a weather-radar image cut into patches \begin{axis}[ at={(58mm,2mm)}, anchor=south west, width=24mm, height=24mm, scale only axis, axis lines=none, enlargelimits=false, xmin=-0.5, xmax=95.5, ymin=-0.5, ymax=95.5, colormap={dictgray}{gray(0cm)=(1); gray(1cm)=(0)}, point meta min=0, point meta max=21, ] \addplot [matrix plot*, mesh/cols=96, point meta=explicit] table [x=px, y=py, meta=rr, col sep=comma] {transformer_radar.csv}; % 16x16-pixel patch boundaries (\pgfplotsinvokeforeach: % a plain \foreach variable would be gone when the % deferred \draw paths are evaluated at \end{axis}) \pgfplotsinvokeforeach{15.5,31.5,47.5,63.5,79.5}{ \draw[black!50, line width=0.3pt] (axis cs:-0.5,#1) -- (axis cs:95.5,#1); \draw[black!50, line width=0.3pt] (axis cs:#1,-0.5) -- (axis cs:#1,95.5); } % course of the Danube; Krems at the image center \addplot [white, line width=0.9pt, no marks] table [x=px, y=py, col sep=comma] {transformer_danube.csv}; \addplot [black, line width=0.35pt, no marks] table [x=px, y=py, col sep=comma] {transformer_danube.csv}; \addplot [only marks, mark=*, mark size=1.1pt, mark options={fill=black, draw=white, line width=0.3pt}] coordinates {(47.5,47.5)}; \node[fill=white, inner sep=0.4pt, font=\fontsize{6}{6}\selectfont] at (axis cs:47.5,56) {Krems}; \draw[black, line width=0.4pt] (axis cs:-0.5,-0.5) rectangle (axis cs:95.5,95.5); \end{axis} \node at (70mm,30mm) {radar-image data point}; \node at (70mm,-4mm) {tokens = patches}; \node at (70mm,-10mm) {(b)}; \end{tikzpicture} \caption{Two data points that consist of tokens. (a) The opening words of the Universal Declaration of Human Rights (United Nations General Assembly, 1948); each word is one %to be more precise, maybe Art. can be cited too token. (b) A weather-radar-based image of the one-hour precipitation around Krems an der Donau, Austria (August 20, 2026; INCA (Integrated Nowcasting through Comprehensive Analysis) analysis, GeoSphere Austria), with $96 \times 96$ pixels of 1 km; darker pixels indicate more precipitation, and thin gray lines cut the image into $16 \times 16$-pixel patches, each of which is one token. The black line, drawn with a white casing so it stays visible over dark pixels, marks the course of the Danube, and the dot at the image center the location of Krems (river course: OpenStreetMap contributors). Data generated by \texttt{pythondemos/transformer.py}} \label{fig_transformer_tokens_dict} \end{figure} The feature vectors (or embeddings) $\featurevec^{(1)}, \ldots, \featurevec^{(\contextlen)} \in \reals^{\featuredim}$ of the $\contextlen$ tokens are stacked into a matrix $\mX = \big(\featurevec^{(1)}, \ldots, \featurevec^{(\contextlen)}\big)^{\top} \in \reals^{\contextlen \times \featuredim}$. Each layer of the original transformer maps such a matrix to a transformed matrix of the same shape, so that layers can be composed freely into a deep ANN (Vaswani et al., 2017, Sect.~3.1). For example, a transformer that translates a sentence transforms the stacked word embeddings, layer by layer, into feature vectors from which the translated words are predicted (Vaswani et al., 2017); transformers also process images by taking patches as tokens (Dosovitskiy et al., 2021). Contemporary large language models (LLMs), such as GPT-3, are transformers trained to predict the next token of a text (Brown et al., 2020). \par The layers of a transformer alternate between two types of transformations of $\mX$ (see Fig.~\ref{fig_transformer_block_dict}). A token-mixing layer maps $\mX$ to a new matrix $\widetilde{\mX}$, each row of which is a weighted sum of linear images of the rows of $\mX$, with weights that can depend on $\mX$ itself; the tokens thereby interact. In the original transformer, token mixing is implemented by multi-head attention, whose output for each token is a weighted sum of the value vectors, the rows of $\mX$ multiplied by a learned matrix (Vaswani et al., 2017, Sect.~3.2). Attention distinguishes transformers from recurrent neural networks (RNNs), which process the tokens of a data point sequentially instead of in parallel (Vaswani et al., 2017, Sect.~1). A position-wise multilayer perceptron (MLP) then transforms each row of $\widetilde{\mX}$ separately (Vaswani et al., 2017, Sect.~3.3). For each of the two transformations, a residual connection adds the input of the transformation to its output, and a subsequent layer normalization standardizes each row of the resulting matrix (Vaswani et al., 2017, Sect.~3.1). \begin{figure}[H] \centering \begin{tikzpicture}[ node distance=4mm, every node/.style={font=\scriptsize}, block/.style={draw, rounded corners=1pt, minimum width=15mm, minimum height=5mm, align=center, inner xsep=2pt}, wide/.style={block, text width=16mm}, addnode/.style={draw, circle, inner sep=0pt, minimum size=4mm}, arr/.style={->, >=stealth, line width=0.5pt}, arrskip/.style={->, >=stealth, line width=0.35pt, densely dashed}, ] \node[coordinate] (in) {}; \node[wide, right=of in] (attn) {multi-head attention}; \node[addnode, right=of attn] (add1) {$+$}; \node[block, right=of add1] (ln1) {layer norm}; \node[wide, right=of ln1] (mlp) {MLP}; \node[addnode, right=of mlp] (add2) {$+$}; \node[block, right=of add2] (ln2) {layer norm}; \node[coordinate, right=of ln2] (out) {}; \draw[arr] (in) -- (attn); \draw[arr] (attn) -- (add1); \draw[arr] (add1) -- (ln1); \draw[arr] (ln1) -- (mlp); \draw[arr] (mlp) -- (add2); \draw[arr] (add2) -- (ln2); \draw[arr] (ln2) -- (out); % residual (skip) connections \draw[arrskip] (in) -- ++(0,7mm) -| node[near start, above] {residual} (add1); \draw[arrskip] (ln1) -- ++(0,7mm) -| node[near start, above] {residual} (add2); \node[left=0mm of in] {input}; \node[right=0mm of out] {output}; \end{tikzpicture} \caption{One transformer block. Each of the two transformations, multi-head attention for token mixing and a position-wise MLP, is bypassed by a residual connection (dashed arrow) whose output is added back and passed through layer normalization. The largest Llama~3 LLM stacks more than 120 of such blocks (Grattafiori et al., 2024, Sect.~3.2)} \label{fig_transformer_block_dict} \end{figure} The matrices produced by successive layers represent the tokens at increasingly abstract levels. This progression is measured by probing: a simple classifier is trained to predict a chosen property from the output of a single layer, and the depth at which the property becomes predictable locates it in the network. In a language transformer, word-level properties are predictable from the output of early layers. One example is a part-of-speech tag, which marks whether a word is a noun, a verb, or an adjective. Relations between tokens far apart in the text become predictable only in deeper layers (Tenney et al., 2019). \par Deep convolutional neural networks (CNNs) show the same progression, from edges and textures in early layers to object parts in deeper ones (Zeiler and Fergus, 2014). Vision transformers develop it too: their early layers split attention between nearby and distant patches, while the deeper layers attend across the whole image (Raghu et al., 2021). In language and vision transformers alike, the most informative representations often arise in the middle layers, whose outputs can outperform those of the final layer as features for other tasks (Skean et al., 2025). \par Whatever the layers represent, computing them is dominated by the token mixing that produces the matrix of each layer, which is computationally expensive. Attention compares every pair of tokens, so its cost grows as $\contextlen^{2}$ with the number $\contextlen$ of tokens (Tay et al., 2022, Sect.~2.4). This quadratic growth motivates alternative techniques for token mixing. Some transformer variants reduce the number of tokens in deeper layers (Tay et al., 2022). Any transformation that combines the rows of $\mX$ can replace attention: an MLP applied across tokens (Tolstikhin et al., 2021), a fixed Fourier transform (Lee-Thorp et al., 2022), or a state-space layer whose cost is linear in $\contextlen$ (Gu and Dao, 2024, Sect.~3.3). \subsection{Cross References} attention, ANN, layer, token, embedding, word embedding, RNN, NLP, LLM, MLP, residual connection, layer normalization, CNN \subsection{References} \begin{enumerate} \item Brown et al. (2020). Language Models are Few-Shot Learners. In: Adv. Neural Inf. Process. Syst., pp.~1877--1901. \item Dosovitskiy et al. (2021). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In: Int. Conf. Learn. Represent. (ICLR). \item Gu and Dao (2024). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. In: Conf. Language Modeling. \item Lee-Thorp et al. (2022). FNet: Mixing Tokens with Fourier Transforms. In: Proc. 2022 Conf. North Amer. Chapter Assoc. Comput. Linguistics: Human Lang. Technologies, pp.~4296--4313. \item Grattafiori et al. (2024). The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783. URL: https://arxiv.org/abs/2407.21783. \item Raghu et al. (2021). Do Vision Transformers See Like Convolutional Neural Networks?. In: Adv. Neural Inf. Process. Syst., pp.~12116--12128. \item Skean et al. (2025). Layer by Layer: Uncovering Hidden Representations in Language Models. In: Proc. 42nd Int. Conf. Mach. Learn. (ICML). \item Tay et al. (2022). Efficient Transformers: A Survey. ACM Computing Surveys, 55(6), pp.~Art.\ no.\ 109. \item Tenney et al. (2019). BERT Rediscovers the Classical NLP Pipeline. In: Proc. 57th Ann. Meeting Assoc. Comput. Linguistics, pp.~4593--4601. \item Tolstikhin et al. (2021). MLP-Mixer: An all-MLP Architecture for Vision. In: Adv. Neural Inf. Process. Syst., pp.~24261--24272. \item United Nations General Assembly (1948). Universal Declaration of Human Rights (UDHR). United Nations, New York. \item Zeiler and Fergus (2014). Visualizing and Understanding Convolutional Networks. In: Proc. European Conf. Comput. Vis. (ECCV), pp.~818--833. \item Vaswani et al. (2017). Attention is all you need. In: Adv. Neural Inf. Process. Syst., pp.~5998--6008. \end{enumerate} \end{document}