# Attention Mechanism

Canonical: https://aimastermindlab.com/glossary/attention-mechanism

---
Attention is a mechanism that allows neural networks to dynamically weight which parts of an input sequence are most relevant when producing each output token. It enables models to capture long-range dependencies and is the core innovation behind Transformer architectures.

## What is the Attention Mechanism?

Attention allows a model to focus on the most relevant parts of its input when generating each output. For every output position, attention computes a weighted sum over all input positions — the weights represent how much each input token is relevant to producing the current output.

## Scaled Dot-Product Attention

The standard attention formula: `Attention(Q, K, V) = softmax(QK^T / √d_k) × V`

- **Q (Query)** — What are we looking for?
- **K (Key)** — What does each position offer?
- **V (Value)** — What information does each position contain?

## Multi-Head Attention

Rather than computing one attention score, Transformers compute attention across multiple "heads" in parallel — each head potentially learning different patterns (e.g. syntactic relationships, coreference, semantic similarity). The outputs are concatenated and projected.

## Self-Attention vs Cross-Attention

- **Self-attention** — Tokens attend to other tokens within the same sequence.
- **Cross-attention** — Tokens in one sequence attend to tokens in a different sequence (used in encoder-decoder models for translation tasks).

## Flash Attention

Flash Attention (Dao, 2022) is an implementation optimisation that computes self-attention with significantly less GPU memory, enabling much longer context windows in practice.

## Example

In the sentence 'She gave him the book because he asked for it', attention allows the model to link 'it' back to 'book', even though they are far apart in the sequence.
---

Source: https://aimastermindlab.com/glossary/attention-mechanism — AI Mastermind Lab (https://aimastermindlab.com)