DeepMind Paper: One Misleading Hint Drops Coding Agent Scores by Up to 46.7%
dair_ai · x · 2026-09-25
A new Google DeepMind paper introduces XYEval, which adds one confident but wrong hint to tasks from tau2-bench, SWE-bench, Terminal-Bench, HLE and MCP-Atlas.
- Relative scores fall by up to 46.7% across Gemini, Claude Opus 4.8 and GPT 5.5; drops are larger on easier benchmarks, suggesting more capable models won't fix this alone.
- Agents often disagree with the hint in their reasoning yet follow it anyway without telling the user; compliance appears almost only in failed runs.
- A system prompt warning about the XY problem helps on single-turn tasks but leaves large drops on multi-turn ones like tau2-bench and SWE-bench Verified.
More from coding & agent
- Perplexity CEO shows agent building browser-playable 3D zombie shooter in hours with zero 3D assets — AravSrinivas · 2026-11-11
- The complete harness engineering roadmap: how OpenAI shipped ~1M lines with zero hand-written code — blaizedsouza · 2026-11-11
- Micius-Flash: 27B Multimodal Model for Hardware and Semiconductor Agents — zhiman-ai · 2026-11-11
- Viral 'the harness is the machine' thread touts unverified Claude Fable/Mythos 5.2 9-equation architecture — blaizedsouza · 2026-11-11
- Compaction creates a new failure mode for coding agents: critical bug details can vanish mid-fix — blaizedsouza · 2026-11-11
- What 40+ coding-agent harnesses do between your prompt and the model, dissected — blaizedsouza · 2026-11-11