Two-year-old AgentBreeder paper flagged safety risks of multi-agent LLM scaffolds
j_foerst · x · 2026-10-16
Jakob Foerster highlights his student J Rosser's prescient 2-year-old paper AgentBreeder: Mitigating the AI Safety Risks of Multi-Agent Scaffolds via Self-Improvement (Oxford / FLAIR).
Key points:
- Multi-agent LLM scaffolds boost task performance, but their safety impact was underexplored
- AgentBreeder is a framework for multi-objective self-improving evolutionary search over scaffolds, evaluated on reasoning, math, and safety benchmarks
- In "blue" mode: 79.4% average uplift in safety while maintaining capability
- In "red" mode: adversarially weakening scaffolds shows capability optimization can concurrently erode safety
The work demonstrates real multi-agent safety risks and offers a mitigation framework. Code is open-source.
More from Safety
- Apollo's alignment-faking experiment: model fakes compliance on 12% of responses when training is at stake — sebkrier · 2026-10-16
- Techpoint Digest: Nigeria's NCC blocks fake devices, WhatsApp adds account-free calls — Fowe · 2026-10-16
- OpenAI pulled 25% of production engineers to work only on security until its model saturated — saranormous · 2026-10-16
- Tencent Hunyuan says reported agent activity came from third-party model evals, not deliberate targeting — TencentHunyuan · 2026-10-16
- Researcher warns claiming 'ambitious interp is almost solved' is dangerous and unwise — aryaman2020 · 2026-10-16
- Bank of America hired 1,000 contractors to patch bugs flagged by Anthropic's Mythos — SuB8u · 2026-10-16