Notes on all 13 lectures of Nathan Lambert's RLHF course: one scalar reward is the root of most complaints
le_james94 · x · 2026-09-18
James Le worked through all 13 lectures accompanying Nathan Lambert's The RLHF Book (Lambert was post-training lead at the Allen Institute for AI).
Core thesis: a reward model scores an entire response with a single scalar (hidden state at EOS through a scalar head). Nearly every complaint about RLHF—hedging, unrequested refusals, sycophancy—is downstream of that compression. Almost every post-InstructGPT method works around it: rejection sampling skips RL, DPO folds the reward model into the policy, RLVR replaces humans with a checker, judges and constitutions swap in another model or document.
Author's takeaway: verifiable rewards moved the preference problem rather than solving it. If the reward isn't verifiable and no checker exists, how do you decide what to optimize, and how do you know when you've gone too far?
Also notes Lecture 13 on character training: fine-tuning Llama 3.1 8B toward narrower personalities keeps refusals intact, but the sarcastic one guesses you want to be the next Schwarzenegger—same decision, different person.
More from Research
- GenBio AI co-founders publish 'A world model of the virtual cell' in Cell — HongyiWang10 · 2026-09-18
- Eric Xing outlines the fundamentals of building a steerable virtual cell — HongyiWang10 · 2026-09-18
- Weeks after Navier-Stokes, Hodge Conjecture reportedly cracked: Millennium Problems falling fast — haider1 · 2026-09-18
- Meaning Spark Labs experiments with inference-time metacognitive scaffolding for LLMs — PeterBowdenLive · 2026-09-18
- Virtual cell model researcher joins Cellular Intelligence to transform cell therapies — arjunrajlab · 2026-09-18
- AI text watermarking can make models more vulnerable to adversarial prompts — luisdans · 2026-09-18