Attention Mechanism

Attention is a mechanism that allows neural networks to dynamically weight which parts of an input sequence are most relevant when producing each output token. It enables models to capture long-range dependencies and is the core innovation behind Transformer architectures.

What is the Attention Mechanism?

Attention allows a model to focus on the most relevant parts of its input when generating each output. For every output position, attention computes a weighted sum over all input positions — the weights represent how much each input token is relevant to producing the current output.

Scaled Dot-Product Attention

The standard attention formula: Attention(Q, K, V) = softmax(QK^T / √d_k) × V

  • Q (Query) — What are we looking for?
  • K (Key) — What does each position offer?
  • V (Value) — What information does each position contain?

Multi-Head Attention

Rather than computing one attention score, Transformers compute attention across multiple "heads" in parallel — each head potentially learning different patterns (e.g. syntactic relationships, coreference, semantic similarity). The outputs are concatenated and projected.

Self-Attention vs Cross-Attention

  • Self-attention — Tokens attend to other tokens within the same sequence.
  • Cross-attention — Tokens in one sequence attend to tokens in a different sequence (used in encoder-decoder models for translation tasks).

Flash Attention

Flash Attention (Dao, 2022) is an implementation optimisation that computes self-attention with significantly less GPU memory, enabling much longer context windows in practice.

Example

In the sentence 'She gave him the book because he asked for it', attention allows the model to link 'it' back to 'book', even though they are far apart in the sequence.

Related terms

  • Transformer The Transformer is a neural network architecture introduced by Google in 2017 that uses self-attention to process sequences in parallel. It is the foundation of virtually all modern large language models, including GPT, Claude, Gemini, and LLaMA.
  • LLM (Large Language Model) A Large Language Model is a neural network trained on massive text corpora to predict and generate text. LLMs like GPT-4, Claude, and Gemini exhibit broad language understanding, reasoning, and generation capabilities that emerge from scale.
  • Context Window The context window is the maximum amount of text (measured in tokens) that an LLM can process in a single inference call. Everything outside the window is effectively invisible to the model. Modern models range from 8K to 2M+ token context windows.