Researcher proposes norm: don't build safety-critical systems you can't red-team
dfrsrchtwts · x · 2026-11-11
A researcher threads an argument for the norm "don't build safety-critical systems you can't red-team." Safety systems default to maximal coverage — monitoring actions, CoTs, internals, training from initialization — but red-teamers are human-AI teams that differ from real AIs in every such dimension, and each difference undermines red-team validity: you'd need to faithfully recreate a scheming model's CoT and internals, or craft adversarial initializations. More coverage means less room for red-teaming to operate.
Related event: Researcher Proposes Rule: Don't Deploy Safety Systems You Can't Red-Team(2 posts)→
More from AGI Musings
- Dormant-neuron backdoor paper raises alarm: open-weight models are backdoorable by anyone — DimitrisPapail · 2026-11-11
- Researcher: Don't Mock Imperfect AI Progress Models — Build Better Ones — austinc3301 · 2026-11-11
- AI risk models are untested best guesses, not overestimates, argues researcher — austinc3301 · 2026-11-11
- Ramez Naam: calibration shows AI progress forecasts systematically overestimate — ramez · 2026-11-11
- Rishi Bommasani, SayashK and Arvind Narayanan Launch Nonprofit STABL for AI Societal Resilience — random_walker · 2026-11-11
- Memento 3 clears all 25 public ARC-AGI-3 levels with rulebook-based self-improvement — hbouammar · 2026-11-11