Dictionary of Applied Machine Learning

attention

Typeset PDF version — the authoritative form of this entry

Attention is a mechanism that models the dependencies between the tokens that make up a data point, such as the words in a sentence or the pixel patches of an image. The idea is to represent the relationship between two tokens by a parameterized weight function whose model parameters are learned from a training set. An attention head then computes a new vector representation for each token as a weighted combination of the value vectors of all tokens. Like the weight function, the value vectors are learned. An attention head acts as a differentiable associative memory: a token uses its query vector to retrieve, through the key vectors of the other tokens, the values most relevant to it. Attention heads capture long-range dependencies between tokens regardless of their positions within a data point, and they are a core component of modern large language models (LLMs).

Definition

Some machine learning (ML) applications involve data points composed of smaller units, referred to as tokens. For example, a sentence consists of words, an image of pixel patches, and a network of nodes. In general, the tokens that constitute a single data point are not independent of one another. Instead, each token of a data point depends on specific other tokens of the same data point. The attention mechanism is a building block of artificial neural networks (ANNs) that captures long-range dependencies between tokens regardless of their positions within a data point.

Probabilistic models provide a principled way of representing and analyzing such dependencies (Blei et al., 2003). Attention mechanisms instead represent these dependencies directly, without specifying a probability distribution over the tokens. They represent the relationship between two tokens $\nodeidx$ and $\nodeidx'$ by an attention weight $f^{(\weights)}(\nodeidx,\nodeidx')$, a parameterized function whose model parameters $\weights$ are learned. The attention weight measures how strongly token $\nodeidx$ attends to token $\nodeidx'$. Practical attention mechanisms differ in their choice of the function $f^{(\weights)}(\nodeidx,\nodeidx')$ and in the empirical risk minimization (ERM) variant used to learn $\weights$. The most widely used choice, described below, is scaled dot-product attention (Vaswani et al., 2017).

Scaled dot-product attention derives three vectors from the embedding vector $\featurevec^{(\nodeidx)} \in \reals^{\featuredim}$ of each token $\nodeidx$ via learned projection matrices: a query (vector) $\vq^{(\nodeidx)} = \mW_Q\, \featurevec^{(\nodeidx)}$, a key (vector) $\vk^{(\nodeidx)} = \mW_K\, \featurevec^{(\nodeidx)}$, and a value (vector) $\vv^{(\nodeidx)} = \mW_V\, \featurevec^{(\nodeidx)}$, where \[\mW_Q, \mW_K, \mW_V \in \reals^{\featuredim \times \featuredim}\] are model parameters learned during training. Intuitively, the query determines what a token "asks for," the key of a token determines what it "advertises," and the value carries the information that is aggregated (cf.\ Bishop and Bishop, 2024; Elhage et al., 2021).

The output of the attention mechanism for token $\nodeidx$ is a weighted sum of the values of all tokens, \[ \vz^{(\nodeidx)} = \sum_{\nodeidx'=1}^{\contextlen} \alpha_{\nodeidx,\nodeidx'}\, \vv^{(\nodeidx')} \text{.} \] Here, $\contextlen$ denotes the number of tokens of the data point, and the coefficient $\alpha_{\nodeidx,\nodeidx'}$ is the attention weight, i.e., the scaled dot-product form of $f^{(\weights)}(\nodeidx,\nodeidx')$. The attention weights are computed from the queries and keys in two steps. First, the attention score between tokens $\nodeidx$ and $\nodeidx'$ is \[ s_{\nodeidx,\nodeidx'} = \frac{\big(\vq^{(\nodeidx)}\big)^{\top} \vk^{(\nodeidx')}} {\sqrt{\featuredim}} \text{.} \] The inner product (also called dot product) $\big(\vq^{(\nodeidx)}\big)^{\top} \vk^{(\nodeidx')}$ measures how well the key of token $\nodeidx'$ matches the query of token $\nodeidx$. Second, the attention weight $\alpha_{\nodeidx,\nodeidx'}$ follows by applying the softmax function to the scores $s_{\nodeidx,\nodeidx'}$ across all tokens $\nodeidx'$: \[ \alpha_{\nodeidx,\nodeidx'} = \frac{\exp(s_{\nodeidx,\nodeidx'})} {\sum_{j=1}^{\contextlen} \exp(s_{\nodeidx,j})} \text{.} \] The softmax function normalizes the scores: the attention weights $\alpha_{\nodeidx,\nodeidx'}$ are nonnegative and sum to one across $\nodeidx' = 1, \ldots, \contextlen$, forming a probability distribution over the tokens. Dividing the inner product by $\sqrt{\featuredim}$ in the attention score counteracts its growth with the dimension $\featuredim $ (Vaswani et al., 2017, Sect. 3.2.1). Indeed, $\big(\vq^{(\nodeidx)}\big)^{\top} \vk^{(\nodeidx')}$ is a sum of $\featuredim $ terms, so without the scaling, a large $\featuredim $ would push the softmax function into a region where its gradients nearly vanish. Fig. 1 illustrates scaled dot-product attention for an eight-token sentence.

Figure 1 of the entry attention
Figure 1: Scaled dot-product attention for the sentence "All human beings are born free and equal." Each token embedding $\featurevec^{(\nodeidx)}$ is projected by learned matrices $\mW_Q$, $\mW_K$, $\mW_V$ into query, key, and value vectors. First, the attention score $s_{\nodeidx,\nodeidx'}$ is computed as the scaled dot product of query $\nodeidx$ and key $\nodeidx'$. Then, the softmax function normalizes these scores into attention weights $\alpha_{\nodeidx,\nodeidx'}$ that sum to one across all positions. Arcs at the top illustrate how token 8 ("equal") attends to earlier tokens; thicker arcs indicate higher attention weights (Vaswani et al., 2017). The token "equal" and its query $\vq^{(8)}$ are highlighted by a darker fill: the depicted weights $\alpha_{8,\nodeidx'}$ involve the query of token 8 but the keys and values of all tokens
Moving from a single token to the whole sequence of tokens, an attention head is a parameterized function $\hypothesis^{(\weights)}$ that maps the input embeddings to the output vectors, \[ \hypothesis^{(\weights)}\big(\featurevec^{(1)}, \ldots, \featurevec^{(\contextlen)}\big) = \big(\vz^{(1)}, \ldots, \vz^{(\contextlen)}\big) \text{,} \] with model parameters $\weights$ comprising the three projection matrices $\mW_Q, \mW_K, \mW_V$. Collecting the output vectors as the rows of a matrix $\mZ$, and the queries, keys, and values as the rows of $\mQ$, $\mK$, $\mV$, $\hypothesis^{(\weights)}$ computes them in the compact form \[ \mZ = \operatorname{softmax}\!\!\left(\frac{\mQ\,\mK^{\top}}{\sqrt{\featuredim}}\right) \mV \text{.} \] Here, the softmax function is applied independently to each row of $\mQ\,\mK^{\top}/\sqrt{\featuredim}$, so that each row of the resulting matrix is a probability distribution over the $\contextlen$ tokens. Since the queries, keys, and values are all derived from the same sequence of tokens, this construction is called self-attention. In practice, several attention heads, each with its own projection matrices, operate in parallel; this is called multi-head attention, and each head can specialize to a different relation between tokens (Vaswani et al., 2017). Computing all pairwise scores $s_{\nodeidx,\nodeidx'}$ requires $\contextlen^{2}$ query–key inner products, one for each ordered pair of tokens.

Attention can be read as a differentiable associative memory, i.e., a content-addressable store (Ramsauer et al., 2021): the keys $\vk^{(\nodeidx')}$ act as addresses, the values $\vv^{(\nodeidx')}$ as stored contents, and a token $\nodeidx$ searches the other tokens with its query $\vq^{(\nodeidx)}$. The inner product $\big(\vq^{(\nodeidx)}\big)^{\top}\vk^{(\nodeidx')}$ measures how well the address $\nodeidx'$ matches the query, the softmax function turns the match scores into retrieval weights, and the output $\vz^{(\nodeidx)}$ is the retrieved content.

Because $\vq^{(\nodeidx)}$ and $\vk^{(\nodeidx')}$ are obtained from different projection matrices, the attention weights are directed: in general, $\alpha_{\nodeidx,\nodeidx'} \neq \alpha_{\nodeidx',\nodeidx}$, so that token $\nodeidx$ attending strongly to token $\nodeidx'$ does not imply the reverse. Directionality arises because asking for information (the query) and advertising it (the key) are distinct roles.

A classical dictionary data structure or hash table returns the single value stored at an exactly matching address (Cormen et al., 2022, Ch. 11). Attention instead returns a weighted average. Moreover, its output $\vz^{(\nodeidx)}$ is a differentiable function of the projection matrices $\mW_Q$, $\mW_K$, and $\mW_V$. In particular, the queries, keys, and values can themselves be learned via gradient-based methods for solving ERM. The same retrieval principle, softened into a weighted average, underlies $k$-nearest neighbors ($k$-NN) methods. It also guides efficient implementations: locality-sensitive hashing restricts each query to nearby keys, reducing the cost of one attention head from $\contextlen^{2}$ query–key comparisons to approximately $\contextlen \log \contextlen$, with $\contextlen$ again the number of tokens (Kitaev et al., 2020).

Fig. 2 visualizes the attention weights $\alpha_{\nodeidx,\nodeidx'}$ that a single head learns on sentences from the Universal Declaration of Human Rights (Assembly, 1948). Trained to reconstruct each masked token from the others, the head gives many query tokens a weight concentrated on a few keys rather than spread uniformly, illustrating the associative-memory reading on this small corpus. The heatmap also shows the directed nature of the weights discussed above: the query token "rights" places its full weight on the key token "beings" ($\alpha_{12,3} = 1.0$), while the query token "beings" places no weight on the key token "rights" ($\alpha_{3,12} = 0.0$).

Figure 2 of the entry attention
Figure 2: Attention weights $\alpha_{\nodeidx,\nodeidx'}$ of one self-attention head for the sentence "All human beings are born free and equal in dignity and rights" (Assembly, 1948, Art. 1). Row $\nodeidx$ is the query token; column $\nodeidx'$ is the searched key token. Darker cells carry more weight: for instance, the query token "all" places nearly all of its weight on the key token "are" ($\alpha_{1,4} = 0.94$), and its remaining weights are close to zero, since the weights of each query sum to one. The head was trained on sentences from the Universal Declaration of Human Rights by masking each token and reconstructing it from the others, so the query must retrieve associated tokens through their keys. Data generated by pythondemos/attention.py.
Synonyms: attention mechanism, attention head.

See also: token, embedding, transformer, softmax function, projection, sequence, natural language processing, large language model, model parameter, empirical risk minimization, dictionary, $k$-nearest neighbors, differentiable.

References

  1. Blei et al. (2003). Latent Dirichlet Allocation. J. Mach. Learn. Res..
  2. Vaswani et al. (2017). Attention is all you need. Adv. Neural Inf. Process. Syst..
  3. Bishop and Bishop (2024). Deep Learning: Foundations and Concepts. Springer Nature.
  4. Elhage et al. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread.
  5. Ramsauer et al. (2021). Hopfield networks is all you need. Int. Conf. Learn. Represent. (ICLR).
  6. Cormen et al. (2022). Introduction to Algorithms. MIT Press.
  7. Kitaev et al. (2020). Reformer: The Efficient Transformer. Int. Conf. Learn. Represent. (ICLR).
  8. Assembly (1948). Universal Declaration of Human Rights (UDHR). United Nations.

Cite this entry

@misc{dictml_attention,
  author = {Jung, Alexander},
  title = {attention},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-08-06},
  url = {https://dictionaryofml.org/terms/attention.html}
}