RLHF (Reinforcement Learning from Human Feedback)

RLHF is the training technique used to align LLMs with human preferences. Human annotators rank model outputs, those preferences train a reward model, and the LLM is then fine-tuned using reinforcement learning to maximise the reward signal. RLHF is how ChatGPT was trained to be helpful and safe.

What is RLHF?

Reinforcement Learning from Human Feedback (RLHF) is the alignment training technique that transformed raw next-token predictors into helpful, instruction-following assistants. It was the key technique behind InstructGPT and subsequently ChatGPT.

The Three-Stage RLHF Pipeline

Stage 1 — Supervised Fine-tuning (SFT) Human annotators write ideal responses to prompts. The model is fine-tuned via standard supervised learning on these demonstrations.

Stage 2 — Reward Model Training Annotators rank multiple model outputs for the same prompt (e.g. "Response A is better than B"). A separate neural network (the reward model) is trained to predict human preference scores.

Stage 3 — RL Optimisation (PPO) The LLM is fine-tuned using Proximal Policy Optimisation (PPO), maximising the reward model's score while staying close to the SFT baseline (via KL divergence penalty).

Limitations of RLHF

  • Reward hacking — Models learn to maximise reward model scores without actually improving, e.g. by producing very long, verbose answers.
  • Annotator disagreement — Human preferences are inconsistent and culturally variable.
  • Scalability — Human annotation is expensive and slow.

Alternatives to RLHF

  • DPO (Direct Preference Optimisation) — Eliminates the separate reward model, directly optimising on preference pairs. Simpler, stabler, and increasingly preferred.
  • Constitutional AI — Anthropic's approach; uses the model itself to critique and revise outputs according to a set of principles.

Example

OpenAI trained InstructGPT via RLHF: for each prompt, multiple GPT completions were generated, labellers ranked them, a reward model learned those preferences, and PPO used the reward model to make GPT more helpful and honest.

Related terms

  • Fine-tuning Fine-tuning is the process of continuing to train a pre-trained model on a smaller, task-specific dataset to specialise its behaviour. It updates the model's weights, making it more accurate for a specific domain, style, or task without training from scratch.
  • LoRA (Low-Rank Adaptation) LoRA is a parameter-efficient fine-tuning technique that trains only a small set of low-rank adapter matrices instead of updating all model weights. It reduces the number of trainable parameters by up to 10,000x while achieving fine-tuning quality comparable to full parameter updates.
  • LLM (Large Language Model) A Large Language Model is a neural network trained on massive text corpora to predict and generate text. LLMs like GPT-4, Claude, and Gemini exhibit broad language understanding, reasoning, and generation capabilities that emerge from scale.
  • DPO (Direct Preference Optimisation) DPO is a simpler alternative to RLHF for aligning LLMs with human preferences. It directly optimises the language model on preference pairs (chosen vs rejected responses) without training a separate reward model or using reinforcement learning.