XYBench Reveals Top LLMs Miss User Misunderstandings in Over 37% of Cases

CMU, Ai2, NVIDIA and Johns Hopkins released XYBench, an 8,115-query benchmark testing whether LLMs can detect misconceptions embedded in user questions. Even the strongest models identify such misunderstandings less than 63% of the time, often answering literally instead of addressing the user's real need.

2026-10-25 ~ 2026-10-27 · 2 related posts

• XYBench: Strongest LLMs Answer Literally and Detect Under 63% of User Misconceptions — MaartenSap · 2026-10-25
• CMU's XYBench: LLMs answer queries literally and reinforce user misconceptions — valentina__py · 2026-10-27