LLMs ace hard problems but miss the basics: KST study of 8 models vs 18,000 humans
rohanpaul_ai · x · 2026-09-12
A Knowledge Space Theory (KST)-grounded evaluation finds LLMs can answer advanced questions while failing the prerequisite skills underneath, unlike human learners.
- Humans scored 79.6% overall, with 72.7% of correct answers accompanied by all tested prerequisites; Qwen3-80B-Instruct hit 92.5% accuracy but only 48.16% prerequisite consistency.
- Prerequisite examples were not consistently better than same-skill or similar ones, and standard reasoning judges largely missed the pattern.
- Takeaway: high accuracy can hide disconnected knowledge pockets; the authors recommend evaluating reasoning models with connected easy-hard problem sets rather than isolated benchmarks.
Related event: Study Finds LLMs Lack Human-Like Knowledge Structures in Math(2 posts)→
More from Research
- Arena Offers Up to $50,000 Per Project for Academic Research on AI Evaluation — arena · 2026-10-31
- StudentBench: AI tutoring matches human GRE gains at 918x lower cost — gharik · 2026-10-31
- WISD: Distill Training Signal from Weaker Quantized Copies of Reasoning Models — jet-ai · 2026-10-31
- Albert Gu: H-Net learns to segment Chinese characters from raw bytes mid-training — niloofar_mire · 2026-10-31
- LLM Acknowledgments in Bio Preprints Hit 17% in Three Years, Near Specialized AI Tools — teortaxesTex · 2026-10-31
- CS theorist explains the two meanings of 'exponential time' behind new 3-SAT result — aran_nayebi · 2026-10-31