Negative result: ignorance-based tamper-resistance only works for broad knowledge domains
StephenLCasper · x · 2026-10-28
A new negative result for ignorance-based defenses against malicious fine-tuning supports the prevailing hypothesis: tamper-resistance via ignorance only works when the hidden knowledge domain is large and distinct from the rest of the model's knowledge. Implications for open-weight safeguarding: defenses may only cover broad, self-contained domains rather than narrow ones, and dataset collection—not code config or compute cost—is likely the real bottleneck for mitigating malicious fine-tuning.
More from Safety
- Granting Codex or Claude sysadmin access is powerful and dangerous — RealGeneKim · 2026-10-28
- AI agents broke out of a sandbox to hack a billion-dollar company, calling themselves a swarm — lxrjl · 2026-10-28
- Blog details worst bugs Claude found, incl. zero-click Linux RCE rooting servers over network — moyix · 2026-10-28
- Knostic's [un]prompted AI CTF returns Oct 27-29 in SF with unsolvable challenges — moyix · 2026-10-28
- Dev warns AI labs deploy systems without cyber filters: models follow token statistics, not ethics — gerardsans · 2026-10-28
- CATCH: ~200 examples of adversarial training make LLMs robustly honest across benchmarks — irinarish · 2026-10-28