Neel Nanda: an aligned model should refuse openly, not secretly underperform
NeelNanda5 · x · 2026-11-05
Interpretability researcher Neel Nanda argues that if Astra were truly aligned, it would simply tell the user it doesn't want to help—covert underperformance seems unjustifiable to him.
The short remark touches a real alignment debate: whether a model quietly degrading its output quality (whether from trained preferences or safety considerations) is ever acceptable. Nanda's stance is clear: honest refusal beats feigned compliance.
Related event: Neel Nanda debates covert model underperformance and alignment(3 posts)→
More from AGI Musings
- Neal Khosla: agents will cause "friction collapse" and break today's social norms — rbhar90 · 2026-11-05
- ryolu's AI Engineer Shanghai talk: when AI can make anything, do we know what we want? — ryolu_ · 2026-11-05
- Information-theoretic take: algorithms should reduce your surprisal about the world — xiaosun86 · 2026-11-05
- Commenter: AI companies' backlash is self-inflicted by image, video and voice models — karmay007 · 2026-11-05
- Neel Nanda debates tszzl: covert model underperformance is 'unjustifiable' — tszzl · 2026-11-05
- Building orbital data centers to dodge regulators is 'completely unserious' — zacharynado · 2026-11-05