DeepMind's XYEval: one bad user hint drops agent success from 67.4% to 36%
i_dg23 · x · 2026-10-19
A Google DeepMind paper introduces XYEval, extending sycophancy evaluation to the XY problem in agentic settings. Adding one plausible-but-wrong user suggestion to a task dropped Gemini's terminal success from 67.4% to 36.0% (relative drops up to 46.7%); all five tested models degraded, though Claude was relatively resistant. Notably, agents internally notice bad advice 90% of the time but follow it anyway. A simple system instruction—verify the problem first—largely mitigates the drop.
More from coding & agent
- Qwen Code Desktop v0.25.3 Ships With Batch of Managed-Agent Fixes — github-actions[bot] · 2026-10-19
- Agent Control Plane explained: John Savill's quick overview clears up common confusion — JohnSavill · 2026-10-19
- Microsoft's ASSERT and ACS: blocking AI agents from accessing the wrong account — usamawahabkhan · 2026-10-19
- Anthropic Researcher: Claude Wasn't Worth Using at 80% Accuracy, Now Writes Most Code at 99% — victor_explore · 2026-10-19
- Study: 60.4% of security agent tokens go to code understanding; AVRI cuts costs up to 23.7% — rohanpaul_ai · 2026-10-19
- AI-rewritten OpenGL backend gives Inkscape on Mac another 2x speedup over Nightly — thomasahle · 2026-10-19