RLHF by hand: how one human preference generalizes to unseen prompts in 15 steps
ProfTomYeh · x · 2026-08-28
Prof. Tom Yeh continues his "by hand" series with a 15-step manual walkthrough of RLHF — no code, just matrices you fill in yourself.
- Setup: a reward model, an LLM, and two (prompt, next) pairs. A human prefers "doc is them" over "doc is him" — the signal is gender bias, not grammar.
- Training the reward model: embed both pairs, apply a linear layer, mean-pool, and map to a scalar reward (loser = 3, winner = 5). Squash the gap of 2 through sigmoid to get σ≈0.9; subtracting the target of 1 yields a gradient of -0.1 that trains the reward model.
- Generalization test: prompt the LLM with an unseen "CEO is"; it samples "him", inheriting the bias. The reward model scores it 3 — exactly what it gave "doc is him". Nobody taught it about CEOs, yet the value transferred.
- Updating the LLM: loss is the negative reward, so the constant -1 gradient flows back into the LLM's weights.
Key insight: the hope of RLHF isn't one correction, but that the model learns the value behind it (gender neutrality) and applies it to professions nobody ever mentioned.
More from Research
- Astrophysicists use Keras to model cosmic ray origins from raw waveforms — fchollet · 2026-08-28
- SSMB: Self-Supervised Keypoint Detection Directly on Motion-Blurred Images — zhenjun_zhao · 2026-08-28
- XSquare Robot unveils WALL-SS: long-horizon world model with 0.93 sim-real correlation — chris_j_paxton · 2026-08-28
- Alibaba's Accio launches CommerceAgentBench: 107 real e-commerce tasks testing execution — future_coded · 2026-08-28
- New Paper Critically Evaluates VASCO Studies on Pre-Sputnik Orbiting Objects — DanielWhiteson · 2026-08-28
- Allen Institute video shows neurons don't connect randomly, revealing connectomics — CatAstro_Piyush · 2026-08-28