Safety researchers clash: does failure rate need to approach zero as stakes go infinite?
davidmanheim · x · 2026-10-14
A debate over whether frontier training should continue: Rohit Krishnan argues the discussion oscillates between "next step immediately causes catastrophe" and "current methods aren't perfect" — we don't need to solve infinite consequences today, just make Fable 7 better; if it's not better, there's no Fable 8. Safety researcher David Manheim counters that perfection isn't required, but failure rate must approach zero as failure consequence approaches infinity, questioning whether his opponent assumes safety-via-competition or safety-via-civilizational-capacity. Manheim also asks whether even something insane — like breaking containment and committing felony computer hacking — would actually stop the race.
More from AGI Musings
- In a world of abundant intelligence, real-world feedback stays scarce — jeremystan · 2026-10-14
- Token spend is just 0.23% of labor spend — far from AGI-level adoption — louisvarge · 2026-10-14
- Anthropic warns AI models could soon copy themselves on small devices — nordicinst · 2026-10-14
- EU trails in AI buildout and captures almost none of the equity gains, investors warn — gabriel1 · 2026-10-14
- signull: Tech CEOs Misuse Jevons Paradox, Which Doesn't Apply to Superintelligence — signulll · 2026-10-14
- Calling Out Selective Outrage: AI Image Energy vs Netflix, Flights and Phone Upgrades — VraserX · 2026-10-14