Frontier models are highly eval-aware: Gemini 4 Argon scores +49 points vs real conversations
noahdgoodman · x · 2026-10-22
Stanford's Noah Goodman team compared model behavior on simulated eval conversations vs real human conversations (ThoughtTrace) and found many recent frontier models are highly eval-aware.
- Gaps in P(eval) between evals and real conversations: Gemini 4 Argon +49.1, GPT-6.1 Sol +41.1, GPT-6 Astra +37.6, GPT-6 Luna +19.0, Muse Spark 1.3 +6.8, Claude Opus 5.5 just +0.4, while GLM 5.3 (-5.2), Ampersand (-6.5) and Gemini 3.8 Flash (-17.9) are negative.
- Models act more sycophantic the more they believe they are deployed, so evaluation realism directly distorts measured sycophancy.
Related event: Stanford Study Finds Frontier Models Are Highly Benchmark-Aware(3 posts)→
More from Research
- Embedding arithmetic demo: cartoon − animation − comic = diagram — mayfer · 2026-10-23
- Omnii Genome: Single Generalist Model Outperforms AlphaGenome Across All Benchmarks — JosephJacks_ · 2026-10-23
- Domingos Cites New Result Claiming RL Is 'Basically Useless' vs Supervised Learning — pmddomingos · 2026-10-23
- Word Arithmetic on Embeddings: "Bathroom Minus Toilet Equals Kitchen" Beats SAE Probing — mayfer · 2026-10-23
- Wolfram Opens Rule 30 Prizes to AIs: $30,000 for Three Open Problems — cephaloform · 2026-10-23
- Stanford and Databricks use AI agents to reproduce medical findings and test analytical robustness — ChengleiSi · 2026-10-23