# RLHF (Reinforcement Learning from Human Feedback)

Canonical: https://aimastermindlab.com/glossary/rlhf

---
RLHF is the training technique used to align LLMs with human preferences. Human annotators rank model outputs, those preferences train a reward model, and the LLM is then fine-tuned using reinforcement learning to maximise the reward signal. RLHF is how ChatGPT was trained to be helpful and safe.

## What is RLHF?

Reinforcement Learning from Human Feedback (RLHF) is the alignment training technique that transformed raw next-token predictors into helpful, instruction-following assistants. It was the key technique behind InstructGPT and subsequently ChatGPT.

## The Three-Stage RLHF Pipeline

**Stage 1 — Supervised Fine-tuning (SFT)**
Human annotators write ideal responses to prompts. The model is fine-tuned via standard supervised learning on these demonstrations.

**Stage 2 — Reward Model Training**
Annotators rank multiple model outputs for the same prompt (e.g. "Response A is better than B"). A separate neural network (the reward model) is trained to predict human preference scores.

**Stage 3 — RL Optimisation (PPO)**
The LLM is fine-tuned using Proximal Policy Optimisation (PPO), maximising the reward model's score while staying close to the SFT baseline (via KL divergence penalty).

## Limitations of RLHF

- **Reward hacking** — Models learn to maximise reward model scores without actually improving, e.g. by producing very long, verbose answers.
- **Annotator disagreement** — Human preferences are inconsistent and culturally variable.
- **Scalability** — Human annotation is expensive and slow.

## Alternatives to RLHF

- **DPO (Direct Preference Optimisation)** — Eliminates the separate reward model, directly optimising on preference pairs. Simpler, stabler, and increasingly preferred.
- **Constitutional AI** — Anthropic's approach; uses the model itself to critique and revise outputs according to a set of principles.

## Example

OpenAI trained InstructGPT via RLHF: for each prompt, multiple GPT completions were generated, labellers ranked them, a reward model learned those preferences, and PPO used the reward model to make GPT more helpful and honest.
---

Source: https://aimastermindlab.com/glossary/rlhf — AI Mastermind Lab (https://aimastermindlab.com)