XYBench Wins EMNLP 2026 Best Resource Paper: LLMs Answer What's Asked, Not What's Needed
hyunw_kim · x · 2026-10-28
The XYBench team (advisor Maarten Sap) won the Best Resource Paper award at EMNLP 2026.
XYBench evaluates whether LLMs can respond pragmatically to queries built on misconceptions — i.e., whether a model flags the wrong premise and addresses what the user actually needs.
Early findings: LLMs tend to answer only what was literally asked, while human experts answer what the asker actually needs. The benchmark is a systematic resource for this 'wrong question' failure mode.
More from Research
- NVIDIA open-sources Flora, a model suite that auto-labels robot videos for training GR00T — qinzytech · 2026-10-28
- Multimodal GraphRAG models missing and contradictory evidence for storm damage assessment — JeremyCMorgan · 2026-10-28
- SIMIT: VLMs self-improve at test time by imagining practice problems, +7.2% on 17 benchmarks — delliott · 2026-10-28
- Mistral sponsors Open Math OS with zero data retention, powering a Kimi K3/DeepSeek math platform — GuillaumeLample · 2026-10-28
- CROP wins EMNLP 2026 Outstanding Paper: conformal certification for reasoning trace prefixes — hanjie_chen · 2026-10-28
- OSU NLP Group Wins Three Paper Awards Same Day at EMNLP 2026 and NeurIPS Workshop — hhsun1 · 2026-10-28