Experiment shows fine-tuned models can detect when activation probes read their internals
Sauers_ · x · 2026-10-18
An experiment by ML researcher Sauers (code and weights open-sourced in the watching GitHub repo) tests whether language models can learn to detect read-only activation probes.
- After fine-tuning the last two layers of Qwen3-8B, the model answers "Yes" when a read-only probe saves one of its layer-14 activations and "No" when it doesn't — 128/128 correct on 64 new prompts, with identical weights and prompt and only the probe as the difference.
- A model trained on two tasks simultaneously also changes an unrelated preference: it says it prefers "owl" when probed and "cat" when not, showing the detection capability spills into unrelated behaviors.
- The author notes untrained models couldn't detect probes, but argues this could plausibly emerge in the wild — just as Claude can tell when it's clearly not in an eval via OOD dialog, models might sense they're not in training from OOD numerical differences — a potential concern for probe-based interpretability and monitoring methods.
More from Safety
- Hackers defrosted hundreds of supermarket fridges — and you're installing smart locks at home? — michalmalewicz · 2026-10-18
- David Sacks: Anthropic blocking 90-99% of distillation proves regulation was never needed — AlexTensor · 2026-10-18
- Redditor proposes AI regulation framework: text detection, face verification, open algorithms — techtotechbytechy · 2026-10-18
- Prompt injection via email can hijack your AI agent — jev-guard adds risk-scored guards — sujingshen · 2026-10-18
- Users press OpenAI on long-delayed Adult Mode a year after it was promised — ___Patrice___ · 2026-10-18
- Viral AI safety report debunked: 'stolen keys' were just a 'test' string — basedjensen · 2026-10-18