# Transformer

Canonical: https://aimastermindlab.com/glossary/transformer

---
The Transformer is a neural network architecture introduced by Google in 2017 that uses self-attention to process sequences in parallel. It is the foundation of virtually all modern large language models, including GPT, Claude, Gemini, and LLaMA.

## What is the Transformer Architecture?

The Transformer, introduced in the landmark paper *"Attention Is All You Need"* (Vaswani et al., 2017), is a neural network architecture built entirely on **self-attention mechanisms**. Unlike previous RNN-based models that processed tokens sequentially, Transformers process all tokens in a sequence simultaneously, enabling massive parallelisation on GPUs.

## Core Components

- **Self-attention** — Each token attends to every other token in the sequence, learning contextual relationships.
- **Multi-head attention** — Multiple attention heads run in parallel, each learning different relationship patterns.
- **Feed-forward layers** — Applied to each position independently after attention.
- **Positional encoding** — Injects information about token position since attention itself is order-agnostic.
- **Layer normalisation and residual connections** — Enable training of very deep networks.

## Encoder vs Decoder

- **Encoder-only** (e.g. BERT) — Good for classification, embeddings, understanding tasks.
- **Decoder-only** (e.g. GPT, LLaMA, Claude) — Good for text generation; the dominant LLM architecture.
- **Encoder-decoder** (e.g. T5, BART) — Good for translation and summarisation.

## Why Transformers Dominate

Before Transformers, RNNs struggled with long-range dependencies (forgetting earlier context). Self-attention solves this by allowing every token to directly attend to every other token, regardless of distance.

## Example

When processing the sentence 'The bank by the river was steep', the Transformer's attention mechanism allows 'bank' to attend strongly to 'river', resolving the ambiguity that a simpler model might miss.
---

Source: https://aimastermindlab.com/glossary/transformer — AI Mastermind Lab (https://aimastermindlab.com)