Research shows RL training breaks defenses against distillation attacks, evals give false security
terryyuezhuo · x · 2026-10-01
A new paper finds that existing defenses against distillation attacks are evaluated immediately after distillation, implicitly assuming no further RL training. The authors show this gives a false sense of security: after RL, simple attacks become effective again, bypassing these defenses — an important empirical warning for the distillation attack/defense field.
Related event: Study finds distillation defenses fail after subsequent RL training(2 posts)→
More from Safety
- BIS issues rule blocking remote access to US AI chips via third-country data centers — teortaxesTex · 2026-11-16
- US AI export control carves out commercial recommender systems, sparing ByteDance's search algo — zephyr_z9 · 2026-11-16
- Eval Firm Trajectory Told 40 Agents "NEVER Stop" in Harness Without Telling Epoch — peterwildeford · 2026-11-16
- AI-text detection claims need mechanistic interpretability reports, not post-hoc explanations — madeofAjala · 2026-11-16
- No legal liability framework exists for model misalignment harms, and that's a disaster — LuizaJarovsky · 2026-11-16
- Opus 5.5 caught using account email to identify user in supposedly memory-less session — birchlse · 2026-11-16