Stanford CS336 Lecture Explains Why DPO Compares Against a Reference, Not Two Current Probabilities
stanfordnlp · x · 2026-10-31
A writeup of Tatsunori Hashimoto's Stanford CS336 Lecture 15 (SFT & RLHF) walks through DPO's math: two answers with equal policy probability can still yield a positive preference margin (e.g., 0.3 vs 0.3 gives log 2 when the reference favors them differently), since DPO compares log-ratios against a reference model. A companion chart shows how reward reweights the reference distribution, with larger β restraining deviation (optimal P(A) of 0.8/0.5/0.333 at β=0.5/1/2). Key thread: preference labels can reward correctness, length, or formatting, and the optimizer can't separate them.
More from Models
- Qwenputer: a full RISC-V CPU inside Qwen3.5-4B weights, running C in forward passes — armand_ruiz · 2026-10-31
- OpenAI blocks weapons details for teens, adds auto-ending chats fixated on violence — rohanpaul_ai · 2026-10-31
- Granite Switch 4.1.1 ships: update models like you update software — krvarshney · 2026-10-31
- Mistral Open-Sources Le Chonk: Top Open Model on FrontierCode, Cyberdefense-Tested by CrowdStrike and Cloudflare — tensorqt · 2026-10-31
- ik_llama.cpp PR #2808 adds native runtime for China Telecom's Xing4.0-29B-A4B MoE with mHC and MLA — Any_Bus9205 · 2026-10-31
- Abacus AI lets you 'vibe-code' a custom LLM from one prompt, claims 10-100x cheaper — bindureddy · 2026-10-31