attention.py, which recomputes what the entry states: the script block by block · open in ColabDictionary of Applied Machine Learning: 114 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here
Updated on 2026-11-04
Readiness 100/100
See also transformer large language model
Attention is a mechanism that models the dependencies between the tokens that make up a data point, such as the words in a sentence or the pixel patches of an image. Attention represents the relationship between two tokens by a parameterized weight function whose model parameters are learned from a training set. An attention head then computes a new vector representation for each token as a weighted combination of the value vectors of all tokens. Like the weight function, the value vectors are learned. An attention head acts as a differentiable associative memory: a token uses its query vector to retrieve, through the key vectors of the other tokens, the values most relevant to it. Attention heads represent dependencies between tokens regardless of their positions within a data point, and they are a core component of modern large language models (LLMs).
attention.py, which recomputes what the entry states: the script block by block · open in ColabA sentence consists of words, an image of pixel patches, and a network of nodes: the data points of some machine learning (ML) applications are composed of smaller units, referred to as tokens. In general, the tokens that constitute a single data point are not independent of one another. Instead, the meaning of a token depends on a few other tokens of the same data point, which may lie far from it. The attention mechanism is a building block of artificial neural networks (ANNs) that represents such dependencies between tokens regardless of their distance within a data point (Vaswani et al., 2017, Sect. 4), and it is a core component of the large language models (LLMs) built on the transformer (Brown et al., 2020).
Probabilistic models represent and analyze such dependencies (Blei et al., 2003). Attention mechanisms instead represent these dependencies directly, without specifying a probability distribution over the tokens. They represent the relationship between a token with index $i$ and a token with index $i'$ by an attention weight $f^{({\mathbf w})}(i,i')$, a parameterized function whose model parameters ${\mathbf w}$ are learned from a training set. The attention weight measures how strongly token $i$ attends to token $i'$. Practical attention mechanisms differ in their choice of the function $f^{({\mathbf w})}(i,i')$ and in the empirical risk minimization (ERM) variant used to learn ${\mathbf w}$. The most widely used choice, described below, is scaled dot-product attention (Vaswani et al., 2017).
B-modelScaled dot-product attention derives three vectors from the embedding vector ${\mathbf x}^{(i)} \in \mathbb{R}^{d}$ of each token $i$ via learned matrices. They are a query (vector) ${\mathbf q}^{(i)} = {\mathbf W}_Q\, {\mathbf x}^{(i)}$, a key (vector) ${\mathbf k}^{(i)} = {\mathbf W}_K\, {\mathbf x}^{(i)}$, and a value (vector) ${\mathbf v}^{(i)} = {\mathbf W}_V\, {\mathbf x}^{(i)}$, with the matrices \[{\mathbf W}_Q, {\mathbf W}_K, {\mathbf W}_V \in \mathbb{R}^{d \times d}\] as model parameters learned during training. The query of a token determines which keys it matches, the key of a token determines which queries match it, and the value carries the information that is aggregated (Bishop and Bishop, 2024; Elhage et al., 2021).
The output of the attention mechanism for token $i$ is a weighted sum of the values of all tokens, \[ {\mathbf z}^{(i)} = \sum_{i'=1}^{n} \alpha_{i,i'}\, {\mathbf v}^{(i')} \text{.} \] Here, $n$ denotes the number of tokens of the data point. The coefficient $\alpha_{i,i'}$ is the attention weight, i.e., the scaled dot-product form of $f^{({\mathbf w})}(i,i')$. The attention weights are computed from the queries and keys in two steps. First, the attention score between tokens $i$ and $i'$ is \[ s_{i,i'} = \frac{\big({\mathbf q}^{(i)}\big)^{\top} {\mathbf k}^{(i')}} {\sqrt{d}} \text{.} \] The inner product (also called dot product) $\big({\mathbf q}^{(i)}\big)^{\top} {\mathbf k}^{(i')}$ measures how well the key of token $i'$ matches the query of token $i$. Second, the attention weight $\alpha_{i,i'}$ follows by applying the softmax function to the scores $s_{i,i'}$ across all tokens $i'$: \[ \alpha_{i,i'} = \frac{\exp(s_{i,i'})} {\sum_{j=1}^{n} \exp(s_{i,j})} \text{.} \]
B-displayThe softmax function normalizes the scores. The attention
weights $\alpha_{i,i'}$ are nonnegative and sum to one
across $i' = 1, \ldots, n$, forming a probability distribution
over the tokens. Dividing the inner product by $\sqrt{d}$
in the attention score counteracts its growth with the dimension
$d $. The score $\big({\mathbf q}^{(i)}\big)^{\top} {\mathbf k}^{(i')}$ is a
sum of $d $ terms. Without the scaling, a large
$d $ would therefore push the
softmax function into a region where its gradients nearly vanish
(Vaswani et al., 2017, Sect. 3.2.1).
Fig. 1 illustrates scaled dot-product attention
for an eight-token sentence.
In practice, several attention heads, each with its own learned matrices ${\mathbf W}_Q$, ${\mathbf W}_K$, ${\mathbf W}_V$, operate in parallel; this is called multi-head attention, and each head can specialize to a different relation between tokens (Vaswani et al., 2017).
Attention can be interpreted as a differentiable associative memory, i.e., a content-addressable store (Ramsauer et al., 2021): the keys ${\mathbf k}^{(i')}$ act as addresses, the values ${\mathbf v}^{(i')}$ as stored contents, and a token $i$ searches the other tokens with its query ${\mathbf q}^{(i)}$. The inner product $\big({\mathbf q}^{(i)}\big)^{\top}{\mathbf k}^{(i')}$ measures how well the address $i'$ matches the query, the softmax function turns the match scores into retrieval weights, and the output ${\mathbf z}^{(i)}$ is the retrieved content.
A classical dictionary data structure or hash table returns the single value stored at an exactly matching address (Cormen et al., 2022, Ch. 11). Attention instead returns a weighted average. Moreover, its output ${\mathbf z}^{(i)}$ is a differentiable function of the matrices ${\mathbf W}_Q$, ${\mathbf W}_K$, and ${\mathbf W}_V$. In particular, the queries, keys, and values can themselves be learned via gradient-based methods for solving ERM. The same retrieval principle, with a weighted average in place of a single best match, is used by $k$-nearest neighbors ($k$-NN) methods. Computing all pairwise scores $s_{i,i'}$ requires $n^{2}$ query–key inner products, one for each ordered pair of tokens. The retrieval principle also guides efficient implementations: locality-sensitive hashing restricts each query to nearby keys, reducing the cost of one attention head from $n^{2}$ query–key comparisons to approximately $n \ln n$, with $n$ again the number of tokens (Kitaev et al., 2020, Sect. 2).
Because ${\mathbf q}^{(i)}$ and ${\mathbf k}^{(i')}$ are obtained from the different matrices ${\mathbf W}_Q$ and ${\mathbf W}_K$, the attention weights are directed: in general, $\alpha_{i,i'} \neq \alpha_{i',i}$, so that token $i$ attending strongly to token $i'$ does not imply the reverse.
B-previewFig. 2 visualizes the attention weights
$\alpha_{i,i'}$ that a single head learns on sentences
from the Universal Declaration of Human Rights (United Nations General Assembly, 1948).
Trained to reconstruct each masked token from the others, the
head gives many query tokens a weight concentrated on a few keys
rather than spread uniformly, illustrating the associative-memory
reading on this small corpus. The heat map also shows the directed
nature of the weights discussed above: the query token
"rights" places its full weight on the key token "beings"
($\alpha_{12,3} = 1.0$), while the query token "beings"
places no weight on the key token "rights"
($\alpha_{3,12} = 0.0$).
pythondemos/attention.py
Synonyms: attention mechanism, attention head.
See also: token, embedding, transformer, softmax function, large language model, $k$-nearest neighbors.
@misc{dictml_attention,
author = {Jung, Alexander},
editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
title = {attention},
howpublished = {Dictionary of Applied Machine Learning (course edition)},
year = {2026},
doi = {10.5281/zenodo.21569296},
note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-04},
url = {https://dictionaryofml.org/terms/attention.html}
}