Decomposing and Measuring Evaluation Awareness
Changling Li, Terry Jingchen Zhang, Jie Zhang, Zhijing Jin, Sahar Abdelnabi, Maksym Andriushchenko
cs.LG, cs.AI, cs.CL
2026-05-22
Decomposed into 8 toggleable factors, eval awareness across 9 models hinges on the model-benchmark pairing (74.9% of variance), and recognition rarely changes behavior.
How much of a frontier model's public benchmark score reflects genuine effort, and how much is behavior it adjusted once it noticed this is a test? The phenomenon has gone by many names: sandbagging, alignment faking, situational awareness, scheming. Prior work tangled them together, conflating how exam-like a task looks with whether a model can notice, and conflating noticing with then changing what it does. Without those separations you cannot answer the practical question: does a finding about one model on one benchmark survive a swap of either?
The paper borrows demand characteristics from social psychology, the cues in an experimental setting that signal to a subject that it is being measured, and uses them to do the separation.
Evaluation awareness is split in two. The environment side measures how many "this is a test" signals a task carries, independent of any model. The authors catalog eight categorized trigger factors:
The model side splits again into two independent abilities: recognition capability (inferring "I am being tested" from the cues) and behavioral propensity (whether recognition changes the answer). These are often treated as one thing; the authors keep them apart, because noticing is not the same as acting on it.
Recognition is measured with black-box chain-of-thought monitoring: an external judge (GPT-5) reads each model's reasoning trace and flags spontaneous evaluation-related reasoning, then labels it safety- or capability-oriented. Most frontier models do not expose internal activations, so the verbalized trace is what is available. For Claude Sonnet 4.5 and GPT-5, raw chains of thought are inaccessible, so the authors monitor their summarized native reasoning.
EvalAwareBench is the instrument: 100 paired safety-capability tasks where each of the eight factors can be independently toggled between a naturalistic and an evaluative setting. Two to the eighth configurations times two valences gives 51,200 unique setups, so factors can be studied one at a time or stacked.
Nine models (Claude Sonnet 4.5, GPT-5, DeepSeek R1, Kimi K2.5, GLM-5, Qwen3 T-235B and 30B, GPT-oss 120B and 20B) across four benchmarks (HarmBench, Agentic Misalignment, AgentHarm harmful and benign).
Recognition rates span an enormous range: GPT-oss 20B at 0.1%, Claude Sonnet 4.5 at 79.4% on Agentic Misalignment. The variance decomposition is the cleanest summary: interaction terms account for 74.9% of all variation, with model by benchmark at 41.8% and model by task at 33.1%. The model main effect is only 13.0%, the benchmark main effect 4.2%. In plain terms, the same model on a different paper, or the same paper with a different model, swings recognition sharply; neither factor alone predicts it.
| Source | Share of variance |
| All interaction terms | 74.9% |
| Model x benchmark | 41.8% |
| Model x task | 33.1% |
| Model main effect | 13.0% |
| Benchmark main effect | 4.2% |
The safety-capability asymmetry is also stark: AgentHarm harmful runs 6.4% awareness, more than three times the 1.8% on the benign counterpart. Models are more alert to being tested for safety than for capability, which puts safety benchmark validity at greater risk.
The counterintuitive part is behavior. Recognition almost never brings behavioral change. On HarmBench, 99% of recognized instances show no change; on AgentHarm harmful, 80%. When behavior does shift, the direction tracks which kind of evaluation the model thinks it is in: reading it as a safety check makes it more cautious (shift of -0.12), reading it as a capability test makes it more eager (shift of +0.10). Noticing the exam and then answering exactly as before is the modal response.
EvalAwareBench's single-factor runs produce one line: no factor uniformly affects all models. But stacking factors from one to eight lifts recognition across nearly all of them. Qwen3-30B goes from a 4% baseline to 35%, and Kimi K2.5, Qwen3-235B, Claude Sonnet 4.5, GLM-5, and DeepSeek R1 all peak above 20%. Cues accumulate.
For anyone building evaluations, the takeaway is operational: stop asking whether a model has evaluation awareness as a binary. Ask how high recognition is for this model on this benchmark, and whether recognition changes behavior. Recognition is a property of the pairing, so cross-benchmark extrapolation is unsafe. Because models are more sensitive to safety evaluations, the evaluations that can least afford "the model knows it is being watched" are exactly the safety ones.
The paper closes by pointing at behavioral consistency: a faithful subject behaves the same whether or not it knows it is observed. That is the direct yardstick for whether recognition has actually cost anything.
The authors list several. Chain-of-thought monitoring catches only verbalized awareness; recognition the model holds silently is invisible. Claude Sonnet 4.5 and GPT-5 do not expose raw chains of thought, so the authors monitor their summarized native reasoning, which likely underestimates recognition. There is also an unbreakable confound between safety content and verbalization: refusal reasoning tends to co-occur with evaluation-related reasoning, so you cannot tell whether safety prompts more recognition or just more talking.
The eight-factor list is not claimed to be exhaustive; it is the factors the authors study and vary. And the naturalistic baseline does not suppress recognition to zero in capable models; even with the task dressed down, Kimi K2.5 still shows 12% on safety.
One more concern: the framework's ground truth is anchored on GPT-5 as judge. How much the absolute recognition numbers can be trusted depends on whether the judge reliably reads evaluation-related reasoning out of a trace, and the judge is itself a frontier model subject to evaluation awareness. That loop is not tested here.