Temperature

Temperature is a hyperparameter that controls the randomness of an LLM's output. A temperature of 0 produces deterministic, predictable responses. Higher values (0.7–1.0+) produce more creative, varied, and sometimes unpredictable text.

What is Temperature in LLMs?

Temperature is a sampling parameter applied to the probability distribution over a model's vocabulary before selecting the next token. It controls the trade-off between determinism and creativity.

How It Works

Before sampling, the model's output logits are divided by the temperature value:

  • Temperature → 0: The distribution becomes highly peaked. The highest-probability token is almost always chosen. Outputs are deterministic and repetitive.
  • Temperature = 1: The distribution is unchanged — tokens are sampled according to their raw probabilities.
  • Temperature > 1: The distribution is flattened. Low-probability tokens become more likely, producing more varied and unpredictable outputs.

Recommended Values by Use Case

Use Case Recommended Temperature
Code generation 0.0 – 0.2
Factual Q&A 0.0 – 0.3
Summarisation 0.3 – 0.5
Chatbot 0.5 – 0.8
Creative writing 0.8 – 1.2
Brainstorming 1.0 – 1.5

Top-p (Nucleus) Sampling

Often used alongside temperature. Top-p restricts sampling to the smallest set of tokens whose cumulative probability exceeds p. Setting top_p=0.9 means only tokens comprising 90% of the probability mass are considered, even after temperature scaling.

Example

Setting temperature=0 for a unit test generator ensures deterministic, syntactically correct code. Setting temperature=1.2 for a story-writing assistant produces more imaginative, unexpected narrative choices.

Related terms

  • LLM (Large Language Model) A Large Language Model is a neural network trained on massive text corpora to predict and generate text. LLMs like GPT-4, Claude, and Gemini exhibit broad language understanding, reasoning, and generation capabilities that emerge from scale.
  • Hallucination Hallucination is when an AI language model generates plausible-sounding but factually incorrect or entirely fabricated information. The model 'hallucinates' facts, citations, names, or events that do not exist, presenting them with false confidence.
  • Tokenization Tokenization is the process of splitting text into smaller units called tokens that an LLM can process. Tokens are typically word fragments, not whole words. Most English text averages about 0.75 words per token, so 1,000 tokens ≈ 750 words.
  • In-Context Learning In-context learning is the ability of LLMs to learn new tasks from examples provided directly in the prompt, without updating model weights. By showing the model input-output pairs (few-shot examples), it generalises to new inputs in the same format.