RLHF debate: conflating reward with what models actually optimize gets riskier as they scale
sofgrey_u47 · x · 2026-09-13
In a debate over whether reward is the optimization target for deeply RL-trained models, the author argues that calling reward the optimization target conflates the training objective with what the model actually learns to optimize for — a distinction that matters more as models become more capable. The original poster had claimed, contra both optimistic and pessimistic versions of "reward is not the optimization target," that reward is indeed the target under deep RL.
Related event: Researchers Debate Whether Reward Is the Optimization Target in Deep RL(9 posts)→
More from Research
- krishnan on autonomous lab agents: criticism changed the next search, not just verdicts — krishnan · 2026-10-31
- Testing agents with explicit rejection records: a cheaper path to verified troubleshooting answers — krishnan · 2026-10-31
- krishnan proposes logging ruled-out troubleshooting hypotheses for future reopening — krishnan · 2026-10-31
- Multi-agent AI lab autonomously researches quantum spins in diamond, revises drifting experiments — krishnan · 2026-10-31
- PI agent rejected 75 proposals in 30 hours; manager banned recurring bad ideas — krishnan · 2026-10-31
- Next 9 proposals explored 7 new directions: rejection became information — krishnan · 2026-10-31