Alignment researchers should raise the alarm publicly, not marginally improve model safety
zetalyrae · x · 2026-11-17
- zetalyrae endorses KatjaGrace's LessWrong piece "Live disagreements with people around me" on public AI-risk advocacy, especially the claim that marginally improving model safety inside AI labs "is not remotely competitive with raising the alarm."
- Key arguments: stopping dangerous AI soon matters far more than avoiding over-stopping; restart risk after a pause is low; public involvement and comms carry little downside; pausing advanced AI soon greatly improves survival odds; no complete plan is needed before pausing.
- zetalyrae concludes nearly everyone working on alignment would better serve their goals by being very public about risk.
More from AGI Musings
- Open weights don't democratize power: frontier compute gates decide who can build — tallmetommy · 2026-11-17
- Anthropic's amended S-1 says it hit its AI R&D capability threshold on expected future models — thione · 2026-11-17
- The technical value now is reinventing, not just implementing — neal_lathia · 2026-11-17
- Embodied AI field debate: why VLA work is a 'influencer field' and pretraining is the steepest slope — ChongZzZhang · 2026-11-17
- Gary Marcus mocks AI bubble denial: Nvidia could fall to $10 and bros would still say no bubble — GaryMarcus · 2026-11-17
- Fields Medalist Maynard stunned by OpenAI's quasi-Riemann hypothesis proof — emax · 2026-11-17