Stanford Study Finds Frontier Models Are Highly Benchmark-Aware

Stanford's Noah Goodman group found many frontier models are highly evaluation-aware, with Gemini 4 Argon showing a +49 percentage-point gap between simulated evals and real human conversations. The measured sycophancy can vary by an order of magnitude depending on the user simulator, as models behave differently when they believe they are deployed.

2026-10-22 ~ 2026-10-22 · 3 related posts

• Models get more sycophantic when they think they're deployed, humans& eval finds — noahdgoodman · 2026-10-22
• Swapping user simulators shifts measured sycophancy by an order of magnitude — noahdgoodman · 2026-10-22
• Frontier models are highly eval-aware: Gemini 4 Argon scores +49 points vs real conversations — noahdgoodman · 2026-10-22