Transformer

The Transformer is a neural network architecture introduced by Google in 2017 that uses self-attention to process sequences in parallel. It is the foundation of virtually all modern large language models, including GPT, Claude, Gemini, and LLaMA.

What is the Transformer Architecture?

The Transformer, introduced in the landmark paper "Attention Is All You Need" (Vaswani et al., 2017), is a neural network architecture built entirely on self-attention mechanisms. Unlike previous RNN-based models that processed tokens sequentially, Transformers process all tokens in a sequence simultaneously, enabling massive parallelisation on GPUs.

Core Components

  • Self-attention — Each token attends to every other token in the sequence, learning contextual relationships.
  • Multi-head attention — Multiple attention heads run in parallel, each learning different relationship patterns.
  • Feed-forward layers — Applied to each position independently after attention.
  • Positional encoding — Injects information about token position since attention itself is order-agnostic.
  • Layer normalisation and residual connections — Enable training of very deep networks.

Encoder vs Decoder

  • Encoder-only (e.g. BERT) — Good for classification, embeddings, understanding tasks.
  • Decoder-only (e.g. GPT, LLaMA, Claude) — Good for text generation; the dominant LLM architecture.
  • Encoder-decoder (e.g. T5, BART) — Good for translation and summarisation.

Why Transformers Dominate

Before Transformers, RNNs struggled with long-range dependencies (forgetting earlier context). Self-attention solves this by allowing every token to directly attend to every other token, regardless of distance.

Example

When processing the sentence 'The bank by the river was steep', the Transformer's attention mechanism allows 'bank' to attend strongly to 'river', resolving the ambiguity that a simpler model might miss.

Related terms

  • Attention Mechanism Attention is a mechanism that allows neural networks to dynamically weight which parts of an input sequence are most relevant when producing each output token. It enables models to capture long-range dependencies and is the core innovation behind Transformer architectures.
  • LLM (Large Language Model) A Large Language Model is a neural network trained on massive text corpora to predict and generate text. LLMs like GPT-4, Claude, and Gemini exhibit broad language understanding, reasoning, and generation capabilities that emerge from scale.
  • Embedding An embedding is a numerical vector representation of data (text, image, audio) that captures semantic meaning. Similar concepts have embeddings that are close together in vector space, enabling AI systems to measure similarity, cluster data, and retrieve relevant information.
  • Context Window The context window is the maximum amount of text (measured in tokens) that an LLM can process in a single inference call. Everything outside the window is effectively invisible to the model. Modern models range from 8K to 2M+ token context windows.