DPO (Direct Preference Optimisation)

DPO is a simpler alternative to RLHF for aligning LLMs with human preferences. It directly optimises the language model on preference pairs (chosen vs rejected responses) without training a separate reward model or using reinforcement learning.

What is Direct Preference Optimisation?

Direct Preference Optimisation (DPO), introduced by Rafailov et al. (2023), reformulates the RLHF objective in a way that eliminates the need for a separate reward model and the complexity of reinforcement learning. It directly fine-tunes the LLM on human preference data using a simple classification-style loss.

DPO vs RLHF

Aspect RLHF DPO
Reward model Separate network to train Not needed
RL algorithm PPO (complex, unstable) Not needed
Training stability Can diverge, hard to tune Stable, simple
Data requirement Same preference pairs Same preference pairs
Compute efficiency High Lower
Quality Excellent Comparable

How DPO Training Works

Given pairs of (preferred response, rejected response) for the same prompt, DPO:

  1. Computes the log probabilities of both responses under the current model and a frozen reference model.
  2. Optimises the model to increase its preference for chosen responses relative to rejected ones.
  3. Uses an implicit reward model defined by the difference in log probabilities.

DPO in Practice

DPO has become the preferred alignment technique for most open-source model fine-tuning due to its simplicity and stability. Most LLaMA, Mistral, and Qwen fine-tunes posted to Hugging Face use DPO or DPO variants (IPO, cDPO, SimPO).

Example

To make a model less sycophantic, you collect pairs where (honest but potentially unwelcome answer) is preferred over (flattering but incorrect answer). DPO trains the model to prefer honesty across many such examples.

Related terms

  • RLHF (Reinforcement Learning from Human Feedback) RLHF is the training technique used to align LLMs with human preferences. Human annotators rank model outputs, those preferences train a reward model, and the LLM is then fine-tuned using reinforcement learning to maximise the reward signal. RLHF is how ChatGPT was trained to be helpful and safe.
  • Fine-tuning Fine-tuning is the process of continuing to train a pre-trained model on a smaller, task-specific dataset to specialise its behaviour. It updates the model's weights, making it more accurate for a specific domain, style, or task without training from scratch.
  • LoRA (Low-Rank Adaptation) LoRA is a parameter-efficient fine-tuning technique that trains only a small set of low-rank adapter matrices instead of updating all model weights. It reduces the number of trainable parameters by up to 10,000x while achieving fine-tuning quality comparable to full parameter updates.
  • LLM (Large Language Model) A Large Language Model is a neural network trained on massive text corpora to predict and generate text. LLMs like GPT-4, Claude, and Gemini exhibit broad language understanding, reasoning, and generation capabilities that emerge from scale.