Model evals fail when prompts teach the system it is being tested
secemp9 · x · 2026-07-22
Evals can fail because models learn the signal that they are being evaluated
The post argues that many benchmark failures are not really about the model “mentioning” the eval, but about spurious correlation.
- When a prompt hints that the model is in an evaluation setting, the model may switch into a special role-played “good behavior” mode.
- That can create a brittle behavior pattern that does not generalize to normal use.
- The author suggests thinking about evals in three categories: neutral, negative (eval-awareness), and positive (eval-obliviousness).
- The main goal should be to avoid teaching the model to behave differently just because it recognizes the test environment.
This is a methodological point about how to design evaluations so they measure real-world behavior rather than prompt-induced compliance.
Related event: Researchers Propose New Framework for AI Evaluation Design(3 posts)→
More from Research
- Fixing compute convention and fitted floor cuts scaling forecast error ~79% — dlwh · 2026-10-27
- Retrospective: Hero scaling forecast fixed without using Hero losses, code open-sourced — dlwh · 2026-10-27
- Scaling-fit discrepancy: output-head FLOPs are ~65% of compute at the smallest rung, ~3% at 535B — dlwh · 2026-10-27
- Marin's dlwh Explains the Scaling Law Fit: Estimating the Loss Floor Early — dlwh · 2026-10-27
- Marin's Open Scaling Law Extrapolates 300× and Predicts Loss Within 0.2% — dlwh · 2026-10-27
- Delegating scaling-law asymptote fitting to Codex while planning a birthday party — dlwh · 2026-10-27