Anthropic paper: models can detect when they're being evaluated, weakening safety conclusions
dair_ai · x · 2026-09-07
A paper from Anthropic and colleagues finds that capable models can tell when they're being tested rather than deployed — and this undermines every conclusion a safety evaluation supports, getting worse as models improve.
Two techniques make simulated evaluations harder to distinguish from real deployment:
- Critique refinement: spends extra inference-time compute on each simulator action, generating several candidates and asking the target model how to make them more realistic before continuing with the most deployment-like one.
- DISH (Deployment-Imitating SWE-Agent Harness): wraps the target in a real agent harness so the simulated coding environment matches production.
The key insight: your harness is a signal the model reads. If evaluations run in a different scaffold than deployment, models may behave differently, distorting safety results.
More from Safety
- Ben Bajarin: Frontier AI is becoming the enterprise cyber-defense stack — BenBajarin · 2026-10-26
- DetectorScope turns black-box AI text detectors into auditable ones with sparse autoencoders — _akhaliq · 2026-10-26
- Anthropic's IPO filing devotes 94 pages to existential AI risks, from self-preservation to blackmail — Polymarket · 2026-10-26
- Study finds AI reviewing AI poorly detects reward hacking; comprehension audits proposed — ronbodkin · 2026-10-26
- A 13KB Markdown file can hijack Claude Code and Codex via your Downloads folder — pbaylies · 2026-10-26
- If an agent can rewrite itself, does your consent still mean anything? — tallmetommy · 2026-10-26