Perplexity

Perplexity is a metric that measures how well a language model predicts a text sample. Lower perplexity means the model assigns higher probabilities to the actual tokens — it is less 'surprised' by the text. It is one of the primary intrinsic evaluation metrics for LLMs.

What is Perplexity?

Perplexity (PPL) is the exponential of the average negative log-likelihood per token that a language model assigns to a text:

PPL = exp(-1/N × Σ log P(token_i | context))

Intuitively: if a model assigns perplexity 10 to a text, it is as uncertain as if it had to choose uniformly among 10 equally likely continuations at each step.

Interpreting Perplexity

  • Lower is better — A model with lower perplexity predicts the text more accurately.
  • A character-level model on English might achieve PPL ~5-20.
  • A word-level model might achieve PPL ~50-200.
  • Domain-specific models typically have lower PPL on in-domain text and higher on out-of-domain text.

Perplexity in Model Evaluation

Perplexity is used to:

  • Compare language models on held-out test sets.
  • Detect data contamination (unusually low PPL on supposedly unseen benchmarks).
  • Evaluate the impact of quantization (higher PPL = more quality loss).
  • Select the best checkpoint during training.

Limitations

Perplexity only measures how well the model predicts token sequences — it doesn't measure factual accuracy, coherence, or usefulness. A model could achieve very low perplexity by memorising training data while still being useless or harmful. That's why downstream task benchmarks (MMLU, HumanEval) and human evaluation are equally important.

Example

A model fine-tuned on medical literature achieves perplexity 12 on a medical textbook held-out set, but perplexity 95 on creative fiction — showing strong in-domain specialisation but worse generalisation.

Related terms

  • LLM (Large Language Model) A Large Language Model is a neural network trained on massive text corpora to predict and generate text. LLMs like GPT-4, Claude, and Gemini exhibit broad language understanding, reasoning, and generation capabilities that emerge from scale.
  • Fine-tuning Fine-tuning is the process of continuing to train a pre-trained model on a smaller, task-specific dataset to specialise its behaviour. It updates the model's weights, making it more accurate for a specific domain, style, or task without training from scratch.
  • Inference Inference is the process of running a trained model on new data to generate predictions or outputs. In LLMs, inference refers to generating a response given a prompt. It is the production phase of the AI lifecycle — distinct from training, which updates model weights.
  • Tokenization Tokenization is the process of splitting text into smaller units called tokens that an LLM can process. Tokens are typically word fragments, not whole words. Most English text averages about 0.75 words per token, so 1,000 tokens ≈ 750 words.