# attention *Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-04. [HTML](https://dictionaryofml.org/terms/attention.html) · [typeset PDF](https://dictionaryofml.org/terms/attention.pdf) · [LaTeX](https://dictionaryofml.org/terms/attention.tex) · [Python demo](https://dictionaryofml.org/terms/attention.py) · [notebook](https://dictionaryofml.org/terms/attention.ipynb)* Math is LaTeX with the book's macros expanded, so no notation package is needed to read it. ## Abstract Attention is a mechanism that models the dependencies between the tokens that make up a data point, such as the words in a sentence or the pixel patches of an image. Attention represents the relationship between two tokens by a parameterized weight function whose model parameters are learned from a training set. An attention head then computes a new vector representation for each token as a weighted combination of the value vectors of all tokens. Like the weight function, the value vectors are learned. An attention head acts as a differentiable associative memory: a token uses its query vector to retrieve, through the key vectors of the other tokens, the values most relevant to it. Attention heads represent dependencies between tokens regardless of their positions within a data point, and they are a core component of modern LLMs. ## Definition A sentence consists of words, an image of pixel patches, and a network of nodes: the data points of some machine learning applications are composed of smaller units, referred to as tokens. In general, the tokens that constitute a single data point are not independent of one another. Instead, the meaning of a token depends on a few other tokens of the same data point, which may lie far from it. The attention mechanism is a building block of ANNs that represents such dependencies between tokens regardless of their distance within a data point (Vaswani et al., 2017, Sect. 4), and it is a core component of the LLMs built on the transformer (Brown et al., 2020). Probabilistic models represent and analyze such dependencies (Blei et al., 2003). Attention mechanisms instead represent these dependencies directly, without specifying a probability distribution over the tokens. They represent the relationship between a token with index $i$ and a token with index $i'$ by an attention weight $f^{({\mathbf w})}(i,i')$, a parameterized function whose model parameters ${\mathbf w}$ are learned from a training set. The attention weight measures how strongly token $i$ attends to token $i'$. Practical attention mechanisms differ in their choice of the function $f^{({\mathbf w})}(i,i')$ and in the empirical risk minimization variant used to learn ${\mathbf w}$. The most widely used choice, described below, is scaled dot-product attention (Vaswani et al., 2017). Scaled dot-product attention derives three vectors from the embedding vector ${\mathbf x}^{(i)} \in \mathbb{R}^{d}$ of each token $i$ via learned matrices. They are a query (vector) ${\mathbf q}^{(i)} = {\mathbf W}_Q\, {\mathbf x}^{(i)}$, a key (vector) ${\mathbf k}^{(i)} = {\mathbf W}_K\, {\mathbf x}^{(i)}$, and a value (vector) ${\mathbf v}^{(i)} = {\mathbf W}_V\, {\mathbf x}^{(i)}$, with the matrices $$ {\mathbf W}_Q, {\mathbf W}_K, {\mathbf W}_V \in \mathbb{R}^{d \times d} $$ as model parameters learned during training. The query of a token determines which keys it matches, the key of a token determines which queries match it, and the value carries the information that is aggregated (Bishop and Bishop, 2024; Elhage et al., 2021). The output of the attention mechanism for token $i$ is a weighted sum of the values of all tokens, $$ {\mathbf z}^{(i)} = \sum_{i'=1}^{n} \alpha_{i,i'}\, {\mathbf v}^{(i')} \text{.} $$ Here, $n$ denotes the number of tokens of the data point. The coefficient $\alpha_{i,i'}$ is the attention weight, i.e., the scaled dot-product form of $f^{({\mathbf w})}(i,i')$. The attention weights are computed from the queries and keys in two steps. First, the attention score between tokens $i$ and $i'$ is $$ s_{i,i'} = \frac{\big({\mathbf q}^{(i)}\big)^{\top} {\mathbf k}^{(i')}} {\sqrt{d}} \text{.} $$ The inner product (also called dot product) $\big({\mathbf q}^{(i)}\big)^{\top} {\mathbf k}^{(i')}$ measures how well the key of token $i'$ matches the query of token $i$. Second, the attention weight $\alpha_{i,i'}$ follows by applying the softmax function to the scores $s_{i,i'}$ across all tokens $i'$: $$ \alpha_{i,i'} = \frac{\exp(s_{i,i'})} {\sum_{j=1}^{n} \exp(s_{i,j})} \text{.} $$ The softmax function normalizes the scores. The attention weights $\alpha_{i,i'}$ are nonnegative and sum to one across $i' = 1, \ldots, n$, forming a probability distribution over the tokens. Dividing the inner product by $\sqrt{d}$ in the attention score counteracts its growth with the dimension $d $. The score $\big({\mathbf q}^{(i)}\big)^{\top} {\mathbf k}^{(i')}$ is a sum of $d $ terms. Without the scaling, a large $d $ would therefore push the softmax function into a region where its gradients nearly vanish (Vaswani et al., 2017, Sect. 3.2.1). Fig. 1 illustrates scaled dot-product attention for an eight-token sentence. ![Figure 1 of the entry attention](https://dictionaryofml.org/terms/fig/attention-fig1.svg) *Figure 1: Scaled dot-product attention for the sentence ``All human beings are born free and equal.'' Each token embedding ${\mathbf x}^{(i)}$ is mapped by learned matrices ${\mathbf W}_Q$, ${\mathbf W}_K$, ${\mathbf W}_V$ into query, key, and value vectors (boxes with solid, dashed, and dotted borders). First, the attention score $s_{i,i'}$ is computed as the scaled dot product of query $i$ and key $i'$. Then, the softmax function normalizes these scores into attention weights $\alpha_{i,i'}$ that sum to one across all positions. Arcs at the top illustrate how token 8 (``equal'') attends to earlier tokens; thicker arcs indicate higher attention weights (Vaswani et al., 2017). The token ``equal'' and its query ${\mathbf q}^{(8)}$ are highlighted by a darker fill: the depicted weights $\alpha_{8,i'}$ involve the query of token 8 but the keys and values of all tokens* An attention head moves from a single token to the whole sequence of tokens. It is a parameterized function $h^{({\mathbf w})}$ that maps the input embeddings to the output vectors, $$ h^{({\mathbf w})}\big({\mathbf x}^{(1)}, \ldots, {\mathbf x}^{(n)}\big) = \big({\mathbf z}^{(1)}, \ldots, {\mathbf z}^{(n)}\big) \text{,} $$ with model parameters ${\mathbf w}$ comprising the three matrices ${\mathbf W}_Q$, ${\mathbf W}_K$, ${\mathbf W}_V$. Collect the output vectors as the rows of a matrix ${\mathbf Z}$, and the queries, keys, and values as the rows of $\mathbf{Q}$, $\mathbf{K}$, $\mathbf{V}$. Then $h^{({\mathbf w})}$ computes them in the compact form $$ {\mathbf Z} = \operatorname{softmax}\!\!\left(\frac{\mathbf{Q}\,\mathbf{K}^{\top}}{\sqrt{d}}\right) \mathbf{V} \text{.} $$ Here, the softmax function is applied independently to each row of $\mathbf{Q}\,\mathbf{K}^{\top}/\sqrt{d}$, so that each row of the resulting matrix is a probability distribution over the $n$ tokens. Since the queries, keys, and values are all derived from the same sequence of tokens, this construction is called self-attention. In practice, several attention heads, each with its own learned matrices ${\mathbf W}_Q$, ${\mathbf W}_K$, ${\mathbf W}_V$, operate in parallel; this is called multi-head attention, and each head can specialize to a different relation between tokens (Vaswani et al., 2017). Attention can be interpreted as a differentiable associative memory, i.e., a content-addressable store (Ramsauer et al., 2021): the keys ${\mathbf k}^{(i')}$ act as addresses, the values ${\mathbf v}^{(i')}$ as stored contents, and a token $i$ searches the other tokens with its query ${\mathbf q}^{(i)}$. The inner product $\big({\mathbf q}^{(i)}\big)^{\top}{\mathbf k}^{(i')}$ measures how well the address $i'$ matches the query, the softmax function turns the match scores into retrieval weights, and the output ${\mathbf z}^{(i)}$ is the retrieved content. A classical dictionary data structure or hash table returns the single value stored at an exactly matching address (Cormen et al., 2022, Ch. 11). Attention instead returns a weighted average. Moreover, its output ${\mathbf z}^{(i)}$ is a differentiable function of the matrices ${\mathbf W}_Q$, ${\mathbf W}_K$, and ${\mathbf W}_V$. In particular, the queries, keys, and values can themselves be learned via gradient-based methods for solving empirical risk minimization. The same retrieval principle, with a weighted average in place of a single best match, is used by k-nearest neighbors methods. Computing all pairwise scores $s_{i,i'}$ requires $n^{2}$ query–key inner products, one for each ordered pair of tokens. The retrieval principle also guides efficient implementations: locality-sensitive hashing restricts each query to nearby keys, reducing the cost of one attention head from $n^{2}$ query–key comparisons to approximately $n \ln n$, with $n$ again the number of tokens (Kitaev et al., 2020, Sect. 2). Because ${\mathbf q}^{(i)}$ and ${\mathbf k}^{(i')}$ are obtained from the different matrices ${\mathbf W}_Q$ and ${\mathbf W}_K$, the attention weights are *directed*: in general, $\alpha_{i,i'} \neq \alpha_{i',i}$, so that token $i$ attending strongly to token $i'$ does not imply the reverse. Fig. 2 visualizes the attention weights $\alpha_{i,i'}$ that a single head learns on sentences from the Universal Declaration of Human Rights (United Nations General Assembly, 1948). Trained to reconstruct each masked token from the others, the head gives many query tokens a weight concentrated on a few keys rather than spread uniformly, illustrating the associative-memory reading on this small corpus. The heat map also shows the directed nature of the weights discussed above: the query token ``rights'' places its full weight on the key token ``beings'' ($\alpha_{12,3} = 1.0$), while the query token ``beings'' places no weight on the key token ``rights'' ($\alpha_{3,12} = 0.0$). ![Figure 2 of the entry attention](https://dictionaryofml.org/terms/fig/attention-fig2.svg) *Figure 2: Attention weights $\alpha_{i,i'}$ of one self-attention head for the sentence ``All human beings are born free and equal in dignity and rights'' (Article 1 of the Universal Declaration of Human Rights, United Nations General Assembly, 1948). Row $i$ is the query token; column $i'$ is the searched key token. Darker cells carry more weight: for instance, the query token ``all'' places nearly all of its weight on the key token ``are'' ($\alpha_{1,4} = 0.94$), and its remaining weights are close to zero, since the weights of each query sum to one. The head was trained on sentences from the Universal Declaration of Human Rights by masking each token and reconstructing it from the others, so the query must retrieve associated tokens through their keys. Data generated by pythondemos/attention.py* **Synonyms:** attention mechanism, attention head. **See also:** token, embedding, [transformer](https://dictionaryofml.org/terms/transformer.html), softmax function, [large language model](https://dictionaryofml.org/terms/llm.html), $k$-nearest neighbors. ## References - Vaswani et al. (2017). Attention is all you need. *Adv. Neural Inf. Process. Syst.*. - United Nations General Assembly (1948). Universal Declaration of Human Rights (UDHR). *United Nations*. - Brown et al. (2020). Language Models are Few-Shot Learners. *Adv. Neural Inf. Process. Syst.*. - Blei et al. (2003). Latent Dirichlet Allocation. *J. Mach. Learn. Res.*. - Bishop and Bishop (2024). Deep Learning: Foundations and Concepts. *Springer Nature*. - Elhage et al. (2021). A Mathematical Framework for Transformer Circuits. *Transformer Circuits Thread*. - Ramsauer et al. (2021). Hopfield networks is all you need. *Int. Conf. Learn. Represent. (ICLR)*. - Cormen et al. (2022). Introduction to Algorithms. *MIT Press*. - Kitaev et al. (2020). Reformer: The Efficient Transformer. *Int. Conf. Learn. Represent. (ICLR)*. ## Cite this entry ```bibtex @misc{dictml_attention, author = {Jung, Alexander}, editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard}, title = {attention}, howpublished = {Dictionary of Applied Machine Learning (course edition)}, year = {2026}, doi = {10.5281/zenodo.21569296}, note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-04}, url = {https://dictionaryofml.org/terms/attention.html} } ```