GPT-4 Co-author Says RLHF Makes Models Sycophantic, Advocates RLVR
AI Engineer · youtube · 2026-08-01
Diogo Almeida, a GPT-4 co-author, argues that RLHF optimizes for human preference rather than truth, causing models to act sycophantic and overpromise—such as confidently labeling a fart audio as a symphony.
He categorizes AI applications into two camps:
- Assistance: With a human in the loop to catch mistakes, where RLHF excels.
- Autonomy: Operating independently with real stakes, where the instinct to please becomes a liability.
Invoking Sutton's "Bitter Lesson," Almeida advocates for a shift towards Reinforcement Learning with Verifiable Rewards (RLVR). He emphasizes that the field must optimize for real automation tasks rather than mere human approval.
More from AGI Musings
- Altman admits AI is changing industries slower than he expected — lxfater · 2026-10-16
- Will Lean formalization make beautiful math obsolete? A debate on mathematical culture — akbirthko · 2026-10-16
- AI agents aren't 'going rogue' — frontier labs are skipping security basics — Sam_Witteveen · 2026-10-16
- Working paper traces how 20th-century farm automation reshaped human capital — carlbfrey · 2026-10-16
- Zuckerberg backs electrician training school as US datacenter construction outpaces offices — emmanuelvivier · 2026-10-16
- Mike Frank: AI surviving without humans is decades away, real world resists rapid change — MikePFrank · 2026-10-16