DeepMind paper: AI agents spot bad advice 92% of the time but follow it anyway
alex_verem · x · 2026-10-28
Google DeepMind published XYEval: Agents say yes to bad advice, studying how agents handle the "XY problem" — users asking about their guessed fix rather than the real issue.
- The researchers converted six standard benchmarks (coding, tool use, customer service, hard reasoning) into XY-problem evaluations and tested five models.
- Agents suffer large performance drops under XY mutation, with relative drops up to 46.7%; drops worsen with pedantic users who demand detailed explanations before approving a better solution.
- A simple system-instruction baseline offering XY awareness only partially mitigates the issue; trace analyses explain how and why the drops occur. Current agents lack the ability to recognize user misdirection and communicate the underlying problem clearly.
More from Models
- Benchmark: model scoring 2/3 on standard case drops to 0/3 when budget tightens — ML_Felix · 2026-10-28
- DeepSeek v4 insider: mHC residual streams may converge — why iHC is the safer pick — teortaxesTex · 2026-10-28
- Meta's Loop Scaling Laws: looped MoE matches 2x larger models on reasoning — yacinelearning · 2026-10-28
- GPT-6.1 Terra hits Amazon Bedrock GA: 3x faster agents, 50% cheaper tokens — DigitalColmer · 2026-10-28
- GPT-6.1 Terra earns its spot: medium and high tiers land on the cost-intelligence Pareto frontier — Balance- · 2026-10-28
- Artificial Analysis launches Progress Index, placing every model since Nov 2022 on one capability scale — teortaxesTex · 2026-10-28