New safety paper builds robust sandbagging model organisms for ELK research
ethayarajh · x · 2026-10-21
A new MATS-stream paper, 'Towards Robust Model Organisms for Eliciting Latent Knowledge,' tackles a key AI-safety problem: extracting information a misaligned model withholds from a weaker overseer (sandbagging).
- Approach: the team trains model organisms to withhold vulnerabilities in a code-scanning task, combining transcript distillation with counter-prompting and tamper-resistance training to instill three properties: Selectivity (sandbags only when it believes the overseer is unaware), Robustness (resists elicitation via prompting and training), and Retention (keeps a knowledge edge over the overseer).
- Results: their strongest organism substantially outperforms prior models on robustness while retaining the other two properties, providing a harsher ELK testbed.
- Code, models, and dataset are open-sourced on Hugging Face.
More from Safety
- RedMonk: MCP and A2A Verify Publishers, Not Safety — Registries Become the Trust Layer — rseroter · 2026-10-21
- Dev complains Anthropic's bio safeguards trigger just from reading a bio paper — robleclerc · 2026-10-21
- Redditor claims uncensored local DeepSeek build answers virus-writing prompt with zero hesitation — Only_Light_9817 · 2026-10-21
- Pristine Models: training image generators on unsafe data without reproducing unsafe concepts — rzhang88 · 2026-10-21
- Interconnects Artifacts 25: 28 new models, DeepSeek V4.1 Flash, and US-China regulatory controls — natolambert · 2026-10-21
- Prompt injection via state input works on decision models too, security researcher demos — wunderwuzzi23 · 2026-10-21