CMU's XYBench: LLMs answer queries literally and reinforce user misconceptions
valentina__py · x · 2026-10-27
A new EMNLP 2026 paper from CMU, Ai2, NVIDIA, and Johns Hopkins introduces XYBench, testing whether LLMs respond pragmatically when users ask the wrong question. The benchmark contains 8,115 misconception-laden queries from Stack Overflow (1,659), manual curation (102), and WikiHow (6,272). A cooperative answer should solve the stated request, flag the misconception, and address the intended goal. Finding: LLMs answer literally, often reinforcing misconceptions instead of steering users to what they actually need.
Related event: XYBench Reveals Top LLMs Miss User Misunderstandings in Over 37% of Cases(2 posts)→
More from Research
- Exponential launches RepoInject and BrowserInject benchmarks for prompt injection robustness — maksym_andr · 2026-10-27
- Redwood: CoT controllability evals are under-elicited; better prompts boost scores 2-3x — dfrsrchtwts · 2026-10-27
- Inverse-designed optical system solves MNIST using BeamZ — S_Conradi · 2026-10-27
- DeCert supervises generative reward models' rationales without reference explanations — Guiquan Sun · 2026-10-27
- VLM harnesses act as compositional generalizers, new blog post shows — a1zhang · 2026-10-27
- Article proposes a Human Genome Project for AI interpretability — ninamiolane · 2026-10-27