Context Window

The context window is the maximum amount of text (measured in tokens) that an LLM can process in a single inference call. Everything outside the window is effectively invisible to the model. Modern models range from 8K to 2M+ token context windows.

What is a Context Window?

The context window defines how much text an LLM can "see" and "remember" in a single request. It includes the system prompt, the user's message, any retrieved documents, conversation history, and the generated response — all counted together toward the limit.

Why Context Window Size Matters

  • Long documents — Analysing a full codebase, legal contract, or research paper requires a large context.
  • Long conversations — Multi-turn chat applications accumulate history quickly.
  • RAG — Retrieving many document chunks requires space in the context window.
  • Few-shot examples — Including many demonstrations for in-context learning consumes tokens.

Context Window Sizes (2026)

Model Context Window
GPT-4o 128,000 tokens
Claude 3.5 Sonnet 200,000 tokens
Gemini 1.5 Pro 1,000,000 tokens
Gemini 1.5 Flash 1,000,000 tokens
LLaMA 3 70B 128,000 tokens
Mistral Large 128,000 tokens

The "Lost in the Middle" Problem

Research shows that LLMs tend to recall information at the beginning and end of their context more reliably than information in the middle. For RAG systems with large contexts, placing the most important retrieved chunks near the beginning or end of the context yields better results.

Managing Long Contexts

Strategies for working within context limits:

  • Chunking and retrieval (RAG) — Only load relevant sections.
  • Summarisation — Compress old conversation history.
  • Sliding window — Maintain a fixed-size window of recent context.

Example

If you paste a 300-page PDF (≈150,000 tokens) into a GPT-4o chat (128K context limit), the beginning of the document gets cut off. Gemini 1.5 Pro's 1M token context could handle the entire document at once.

Related terms

  • Tokenization Tokenization is the process of splitting text into smaller units called tokens that an LLM can process. Tokens are typically word fragments, not whole words. Most English text averages about 0.75 words per token, so 1,000 tokens ≈ 750 words.
  • LLM (Large Language Model) A Large Language Model is a neural network trained on massive text corpora to predict and generate text. LLMs like GPT-4, Claude, and Gemini exhibit broad language understanding, reasoning, and generation capabilities that emerge from scale.
  • RAG (Retrieval-Augmented Generation) RAG is a technique that combines an LLM with an external knowledge retrieval system. Instead of relying solely on its training data, the model retrieves relevant documents at query time and uses them as context to generate more accurate, up-to-date responses.
  • In-Context Learning In-context learning is the ability of LLMs to learn new tasks from examples provided directly in the prompt, without updating model weights. By showing the model input-output pairs (few-shot examples), it generalises to new inputs in the same format.