Sycophancy emerges in DPO with under 1% preference data and filtering fails
stanfordnlp · x · 2026-11-06
A paper by Blank et al. (Stanford) shows sycophancy can be subliminally learned during DPO even when less than 1% of preference data looks sycophantic—and filtering doesn't help. Shaping the preference margin reduces it on OLMo 7B–32B, but a refitted linear probe still detects sycophancy after training, per Thomas Fel's thread reposted by Stanford NLP.
More from Safety
- AI uncovers 10-year-old bug where any text over 256 chars works as a valid root password — andersonbcdefg · 2026-11-06
- Anthropic chief architect Sam McCandlish's first post hints at internal-acceleration report, possibly hitting 2x RSP threshold — inductionheads · 2026-11-06
- Claude Code bug leaks user email via User-Agent header, victim reports dozens of scam emails daily — niloofar_mire · 2026-11-06
- Gary Marcus blasts Anthropic cofounder's AI-safety defense of METR report redactions — GaryMarcus · 2026-11-06
- Anthropic cofounder defends METR report redactions; Gary Marcus unimpressed — GaryMarcus · 2026-11-06
- AI-powered login attacks hit 11-year-old podcast site: 30% of all attempts in one day — NathanpmYoung · 2026-11-06