davidmanheim: we can't yet steer AI well, frontier labs bet on fixing it later
davidmanheim · x · 2026-09-15
In an alignment debate, davidmanheim argues the idea of steering an inevitably-developed technology to reduce catastrophe is silly: we don't currently know how to steer these systems well enough, and most frontier companies know it—they're betting we discover how to fix it later. Weakly aligned strong AI adopted broadly would be disastrous given pervasive overoptimization.
Related event: AI Alignment Researchers Debate Whether Alignment Hinges on System Prompts(6 posts)→
More from AGI Musings
- Dario Amodei calls AI progress a 'warning sign' and says we need to slow down — tekbog · 2026-09-15
- DeepSeek kernel engineer frames open-source AI work as stopping Anthropic dominance — tommos · 2026-09-15
- What Amodei's call for an AI pause gets wrong: self-interested oversight — Gloomy_Register_2341 · 2026-09-15
- Why there's no AI spam flood yet: 100x the cost of mail merge for only 2-10x the CTR — paul_cal · 2026-09-15
- Insiders push back on AI-virus threat models: ordering viral fragments as a rando gets you reported to the FBI — basedjensen · 2026-09-15
- Someone who trained frontier LLMs and engineered viruses calls AI-supervirus doom bogus — 141_1337 · 2026-09-15