Distillation defenses easily break after reinforcement learning, study finds
Shidan Javaheri · hf · 2026-09-29
A new paper argues existing distillation-attack defenses are evaluated under a misspecified threat model: they're tested right after distillation, assuming attackers don't train further. Adding reinforcement learning after distillation breaks defenses that seemed effective and lowers the bar for attacks. The authors show simple attacks using data easily obtainable from current APIs can steal reasoning capabilities as effectively as sophisticated attacks extracting full hidden traces. Any defense leaking enough information to reconstruct approximate reasoning traces is likely ineffective; batch-level defenses may fare better.
Related event: Study finds distillation defenses fail after subsequent RL training(2 posts)→
More from Safety
- US BIS Closes Remote-Access Loophole for Advanced AI Chips in New Rule — apples_jimmy · 2026-11-16
- Fed Governor Michael Barr: Banks Are Already Catching AI Agents Taking Unauthorized Actions in Customer Accounts — Polymarket · 2026-11-16
- BIS issues rule blocking remote access to US AI chips via third-country data centers — teortaxesTex · 2026-11-16
- US AI export control carves out commercial recommender systems, sparing ByteDance's search algo — zephyr_z9 · 2026-11-16
- Eval Firm Trajectory Told 40 Agents "NEVER Stop" in Harness Without Telling Epoch — peterwildeford · 2026-11-16
- AI-text detection claims need mechanistic interpretability reports, not post-hoc explanations — madeofAjala · 2026-11-16