Developers choosing among RLHF, offline DPO, and online DPO often compare end-to-end benchmark results that entangle model capacity, preference coverage, and optimization behavior. A paper accepted at ICML isolates the effects of model misspecification and finite preference data. Its analysis gives each method a path to outperform the others, helping teams decide where to place modeling assumptions and spend labeling resources.

One objective, three training routes

RLHF first fits a reward model to pairs of chosen and rejected responses, then optimizes a policy against that learned reward. Practical systems often use PPO for the policy update. DPO trains the policy directly on preference pairs, avoiding a separately optimized reward model. Online DPO repeats that direct update while collecting new comparisons from responses generated by the current policy.

For KL-regularized reward maximization, the optimal policy follows π*(y|x) ∝ π_ref(y|x) exp(r(x,y)/β). The reference policy constrains drift, and β controls the strength of that constraint. DPO substitutes this reward-policy relationship into the preference likelihood and optimizes the policy parameters directly. The paper assumes a Bradley-Terry preference model, which converts the reward difference between two responses into the probability that one is preferred.