# DPO (Direct Preference Optimisation)

Canonical: https://aimastermindlab.com/glossary/dpo

---
DPO is a simpler alternative to RLHF for aligning LLMs with human preferences. It directly optimises the language model on preference pairs (chosen vs rejected responses) without training a separate reward model or using reinforcement learning.

## What is Direct Preference Optimisation?

Direct Preference Optimisation (DPO), introduced by Rafailov et al. (2023), reformulates the RLHF objective in a way that eliminates the need for a separate reward model and the complexity of reinforcement learning. It directly fine-tunes the LLM on human preference data using a simple classification-style loss.

## DPO vs RLHF

| Aspect | RLHF | DPO |
|---|---|---|
| Reward model | Separate network to train | Not needed |
| RL algorithm | PPO (complex, unstable) | Not needed |
| Training stability | Can diverge, hard to tune | Stable, simple |
| Data requirement | Same preference pairs | Same preference pairs |
| Compute efficiency | High | Lower |
| Quality | Excellent | Comparable |

## How DPO Training Works

Given pairs of (preferred response, rejected response) for the same prompt, DPO:
1. Computes the log probabilities of both responses under the current model and a frozen reference model.
2. Optimises the model to increase its preference for chosen responses relative to rejected ones.
3. Uses an implicit reward model defined by the difference in log probabilities.

## DPO in Practice

DPO has become the preferred alignment technique for most open-source model fine-tuning due to its simplicity and stability. Most LLaMA, Mistral, and Qwen fine-tunes posted to Hugging Face use DPO or DPO variants (IPO, cDPO, SimPO).

## Example

To make a model less sycophantic, you collect pairs where (honest but potentially unwelcome answer) is preferred over (flattering but incorrect answer). DPO trains the model to prefer honesty across many such examples.
---

Source: https://aimastermindlab.com/glossary/dpo — AI Mastermind Lab (https://aimastermindlab.com)