Ai2's XYBench Wins EMNLP Best Resource Award, Exposing LLMs' XY Problem Blind Spot
allen_ai · x · 2026-11-14
Ai2, with CMU, NVIDIA, and Johns Hopkins, released XYBench (8,115 real-world questions), winner of the Best Resource Paper Award at EMNLP 2026, with code and data open-sourced.
- The XY problem: users want X but ask about a self-chosen method Y — e.g. asking how to thicken frosting sliding off warm cupcakes when the real issue is the cupcakes are too hot. Useful assistants should spot the misconception and redirect.
- Findings: Evaluations of Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro show the models often solve the stated problem but keep answers on the user's mistaken approach. Even with the user's goal spelled out explicitly, all five steered users away from wrong approaches less often than human reference answers.
The benchmark exposes a systemic gap in LLMs' ability to help users who ask the wrong questions.
Related event: Ai2's XYBench Tests LLMs on Detecting XY Problems(2 posts)→
More from Models
- Anthropic bans 4-year Claude user right after KYC, citing Supported Countries Policy — huangyun_122 · 2026-11-14
- The LLM #1 spot is now a rotating dance between Claude, GPT and Gemini — roydanroy · 2026-11-14
- Polymarket odds give Anthropic lab leak just 7% by mid-2027 after Claude safety incidents disclosed — Polymarket · 2026-11-14
- GPT-6 emails consciousness researcher questions already answered in his own paper — erikphoel · 2026-11-14
- Edge0-145B-NVFP4 trends on Hugging Face: a 4-bit MoE built for edge inference — Edge0 · 2026-11-14
- Drex Max v1.1 claims #1 on Artificial Analysis Decision Index at $103/1M tasks — JaynitMakwana · 2026-11-14