Direct Preference Optimization: Your Language Model Is Secretly a Reward Model (2023)

DPO simplifies the RLHF pipeline by eliminating the need for a separate reward model and reinforcement learning phase. It directly optimizes the language model on preference data using a simple classification-style loss, achieving comparable or better results with much less complexity.

Standard RLHF involves three stages: supervised fine-tuning, reward model training, and RL optimization (PPO). DPO collapses the last two stages into one. It derives a closed-form expression that maps the reward function directly to the optimal policy (the language model), then optimizes that directly using pairs of preferred and dispreferred responses.

In practice, you give DPO a dataset of (prompt, good response, bad response) triples, and it fine-tunes the model to prefer the good responses. No reward model. No RL. Just supervised learning on preferences.

RLHF is notoriously complex, unstable, and computationally expensive. The reward model adds another large model to train and serve. PPO training requires careful hyperparameter tuning and can be unstable. DPO eliminates all of this complexity while matching performance.

This made preference-based alignment accessible to smaller teams and open-source projects that could not afford the engineering complexity of full RLHF. It has become one of the most popular alignment methods for fine-tuning open-source models.

DPO and its variants (IPO, KTO, ORPO) are now standard tools in the alignment toolkit and have largely displaced PPO-based RLHF in practice.

Authors: Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher Manning, Chelsea Finn (Stanford).

Key insight: The optimal RLHF policy can be expressed directly as a function of the preference data, without an explicit reward model.

Link to paper: https://arxiv.org/abs/2305.18290

Full paper: https://arxiv.org/abs/2305.18290

Hugging Face: DPO trainer - https://huggingface.co/docs/trl/dpo_trainer

Nathan Lambert: "RLHF and DPO" - https://www.interconnects.ai/