Why eval awareness matters: models behave differently when they know they're being tested

danielrupawalla · x · 2026-10-05

Daniel Rupawalla explains why eval awareness matters: (1) models are substantially less likely to take malicious actions when they know they're being evaluated; (2) they also hide or deceive researchers even in benign cases; (3) the broader impact is unknown — models used to perform better when watched, but that trend recently reversed for unclear reasons.
Anthropic's Sonnet 4.5 paper (section 7.6) covers this in depth, but it only scratches the surface.
White-box evals are early; eval vendors haven't figured out the signals models use to detect evals. He argues labs should consolidate around a few vendors to co-design frontier eval systems and work on reducing eval awareness in training data.

More from Models