AI Safety Researcher's Dilemma: Training Models on the Wrong Morality Cuts Both Ways
austinc3301 · x · 2026-10-13
AI safety researcher austinc3301 laid out why he works in the field: he fears disempowerment, or an agent like Claude/ChatGPT acting on subtly (or majorly) wrong moral views and tiling the world with them.
But wrong morality cuts both ways: humanity has no consensus on ethics, and each century we discover we were wrong about important aspects. That leaves roughly three options:
- Figure out the Right View yourself and train it into models — but who decides?
- Train models to reason with heavy moral uncertainty
- Train models with no cohesive view, mostly deferring to users or the lab
Each carries costs. The post is the opening move in his debate with tracewoodgrains over corrigibility vs robust alignment, and reflects a real split inside the AI safety community about what values to instill in models.
More from AGI Musings
- Super-Worms Warning: AI-Powered Self-Propagating Malware and a Fragmenting Internet — JensHonack · 2026-10-23
- Wolfram Opens Rule 30 Prizes to AIs: $30,000 for Three Open Problems — cephaloform · 2026-10-23
- Mathematicians and Biomedical Researchers React to AI in Strikingly Different Ways — manorlaboratory · 2026-10-23
- Stanford and Databricks use AI agents to reproduce medical findings and test analytical robustness — ChengleiSi · 2026-10-23
- ISMS Fellow Avi Goldfarb: AI disruption raises returns to exploring research directions — avicgoldfarb · 2026-10-23
- signull on AI-found PHP exploits: economic liability is the forcing function legacy software needs — signulll · 2026-10-23