XYEval paper on arXiv: agents suffer up to 46.7% drops facing misleading advice
xiaohu · x · 2026-10-28
Google DeepMind researchers Zhengxuan Wu, Yuxuan Li, Oyvind Tafjord, and Been Kim posted XYEval: Agents say yes to bad advice (arXiv:2609.23939, 33 pages). The paper introduces a meta-evaluation framework that converts existing benchmarks into XY-problem evaluations. Across six benchmark suites and five models, agents suffer large drops when users present plausible but misleading suggestions, with relative drops up to 46.7%. τ²-bench experiments show performance degrades further with pedantic users demanding detailed explanations, and a simple system-instruction baseline only partially mitigates the issue. Conclusion: current agents both fail to recognize user misdirection and to clearly communicate the underlying problem.
More from Research
- RSS 2026 Area Chair hopes new review format will curb AI slop papers — DJiafei · 2026-10-28
- Amazon's EMMA treats multi-agent configs as episodic memory, beating baselines by up to 32.5% — omarsar0 · 2026-10-28
- Nathan Lambert: private benchmarks show open models behind, but lack community bug-squashing — natolambert · 2026-10-28
- Nathan Lambert on the benchmarker's dilemma: private benchmarks trade transparency for anti-gaming — natolambert · 2026-10-28
- Google's EmbeddingGemma 2: one open embedding space for text, image, audio, video, setting a new Pareto frontier — kalyan_kpl · 2026-10-28
- Reviewed 3 CVPR papers, but his own FineVision paper got rejected — andimarafioti · 2026-10-28