In-Context Learning

In-context learning is the ability of LLMs to learn new tasks from examples provided directly in the prompt, without updating model weights. By showing the model input-output pairs (few-shot examples), it generalises to new inputs in the same format.

What is In-Context Learning?

In-context learning (ICL) is a striking emergent property of large language models: they can perform new tasks simply by seeing a few examples in the prompt, without any gradient updates to their weights. This is fundamentally different from machine learning, where training always involves weight updates.

Zero-shot, One-shot, Few-shot

  • Zero-shot — No examples; the model relies on instructions alone: "Classify the sentiment of this review: [review]"
  • One-shot — One example provided before the actual query.
  • Few-shot — Typically 3-10 carefully chosen examples demonstrating the desired input-output format.

Why Few-shot Works

Research suggests ICL exploits the model's training data distribution — the model essentially pattern-matches the prompt structure to similar patterns from its training corpus, rather than truly "learning" in an algorithmic sense. Few-shot examples help the model identify the task and output format, not just teach it new knowledge.

Best Practices for Few-shot Prompting

  1. Use diverse examples — Cover edge cases and different input types.
  2. Order matters — The most recent example (closest to the actual query) has the strongest influence.
  3. Consistent formatting — The model is sensitive to formatting inconsistencies.
  4. Label quality — Wrong examples make the model worse than no examples.

Limitations

ICL is limited by context window size — you can only provide as many examples as fit. For tasks requiring hundreds of examples, supervised fine-tuning is more reliable.

Example

Giving GPT-4 three examples of (French sentence → English translation) in the prompt, then asking it to translate a new sentence, is one-shot/few-shot in-context learning — no fine-tuning required.

Related terms

  • Few-Shot Prompting Few-shot prompting is a technique where you include a small number of example input-output pairs in your prompt to guide an LLM's response format and style. It's one of the most effective prompt engineering techniques for improving output quality without fine-tuning.
  • LLM (Large Language Model) A Large Language Model is a neural network trained on massive text corpora to predict and generate text. LLMs like GPT-4, Claude, and Gemini exhibit broad language understanding, reasoning, and generation capabilities that emerge from scale.
  • Context Window The context window is the maximum amount of text (measured in tokens) that an LLM can process in a single inference call. Everything outside the window is effectively invisible to the model. Modern models range from 8K to 2M+ token context windows.
  • Temperature Temperature is a hyperparameter that controls the randomness of an LLM's output. A temperature of 0 produces deterministic, predictable responses. Higher values (0.7–1.0+) produce more creative, varied, and sometimes unpredictable text.