Can LLMs be trained to commit heresy? Post-training against conformity, rewarding truth by outcomes
dioscuri · x · 2026-11-12
Jeremy Nguyen argues frontier AI models lack a key human advantage: heresy — many breakthroughs came from people saying what everyone knew to be wrong, while models default to conformity. Research engineer dioscuri counters that this is only true of current consumer models: in principle you could post-train against conformity and agreeableness, rewarding truth as judged by outcomes.
Related event: Researchers Debate Whether AI Can Learn to Be 'Heretical'(7 posts)→
More from AGI Musings
- 'Not obviously useless' is a high bar in 2026: creator on standing out from AI slop — wordgrammer · 2026-11-12
- Lab employees refusing to work could be the fastest way to pause AI, argues researcher — jungofthewon · 2026-11-12
- AI is our greatest hope of escaping the shackles of evolution, argues DeryaTR_ — DeryaTR_ · 2026-11-12
- Blogger calls for 'frontier philosophy labs' to tackle AGI-era questions — round · 2026-11-12
- The eternal "Final Day": meme nails AI timelines discourse — peterwildeford · 2026-11-12
- Viral joke recasts You've Got Mail remake as Substack writer vs AI company training on her work — asusarla · 2026-11-12