ListQA benchmark: best LLM scores under 37% on list-formatted factual recall

_reachsumit · x · 2026-10-15

ListQA (arXiv 2610.19165) is a new benchmark testing how well LLMs recall multiple facts from parameters and compose them into structured lists.

• 9,045 human-curated, cross-validated questions (1.7% audited error rate) requiring 3–10 element lists, from flat to hierarchical with sub-attributes, across 8 categories and 60+ countries.
• Evergreen design: explicit temporal constraints or historically immutable facts keep ground truth valid without updates.
• Evaluation frames element matching as a linear sum assignment problem with LLM-based semantic grading, jointly capturing accuracy, hallucination, and format compliance.
• Benchmarking 34 models (8 families, 42 configs): the best standard-inference model scores below 37% LLM-Judge-F1; multi-faceted questions are harder across the board; providing the source Wikipedia page lifts scores dramatically — models can organize facts but struggle to retrieve them from parameters.

Original post →

More from Research

• Domingos Cites New Result Claiming RL Is 'Basically Useless' vs Supervised Learning — pmddomingos · 2026-10-23
• Wolfram Opens Rule 30 Prizes to AIs: $30,000 for Three Open Problems — cephaloform · 2026-10-23
• Stanford and Databricks use AI agents to reproduce medical findings and test analytical robustness — ChengleiSi · 2026-10-23
• EMNLP paper: diffusion LMs internally encode latent denoising timestep representations — gsarti_ · 2026-10-23
• KU Leuven and VIB open two Group Leader professorships in AI for Biology, start 2027 — jpillowtime · 2026-10-23
• Subspace Tuning curbs LLM catastrophic forgetting by freezing principal singular directions — scottniekum · 2026-10-23

Research channel →