Bengio-backed argument: post-training AI safety defenses are brittle, build safety in from scratch
juliusadml · x · 2026-11-20
A retweet of Pietro Barbiero and Guide Labs' argument, citing Bengio et al.'s paper "Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?" (arXiv:2502.15657).
- Post-training defenses are brittle: Guide Labs found defenses added after training only work against hacks they've seen — a seemingly harmless change to one test weakened nearly every approach tried.
- Post-hoc concept search is costly: locating a concept (e.g. "hack") in a trained model's representations may cost more than training itself and guarantees nothing about whether the concept exists; interpretable concept-based models expose identifiable concepts by construction.
- Conclusion: safety must be designed in from training time — post-training countermeasures won't scale as capabilities grow. Barbiero endorses LawZero and Guide Labs as on the right track.
- Paper context: Bengio and 12 co-authors analyze deception and loss-of-control risks from agency-driven training, proposing non-agentic "Scientist AI" as a safer building block.
More from Safety
- Persimmon built a safety framework from scratch for human-like user models — gharik · 2026-11-21
- Plaid stock plunges 57% after cyber attack; investor sees AI SOC as growth driver — thedealdirector · 2026-11-21
- Tencent Hy4 Ships First Standalone Safety System Card With Third-Party Evals — wfithian · 2026-11-21
- Arena Alignment Index update: OpenAI sweeps top seven, Claude Fable 5.2 debuts at #8 — arena · 2026-11-21
- Security & Web3 Firewall MCP Server Hits Official Registry: Detects Tool-Poisoning and Rug-Pull Drift — ezelech · 2026-11-21
- Third-party AI safety testing takes hold in China with FangcunLeap's ~40-person team — typewriters · 2026-11-21