Study: LLMs' math knowledge lacks coherent structure despite coherent accuracy
dair_ai · x · 2026-09-09
A new paper tests whether LLMs' mathematical knowledge has coherent structure or just coherent accuracy, using Knowledge Space Theory — where mastering a concept requires mastering its prerequisites — as a normative standard, evaluating eight open and closed models against real human learners.
Key findings:
- Models frequently violate knowledge dependencies, answering dependent questions correctly while failing their prerequisites, and they fail to leverage related knowledge supplied in context to improve on dependent questions.
- The eight models show low overlap in their knowledge distributions, sharing no consistent structure with each other.
Implication: LLM math ability looks like fragmented accuracy rather than a prerequisite-based knowledge system — a caveat for using LLMs to advance mathematics.
More from Models
- MagicAILabs claims new recipe matches DeepSeek V4 Pro pretraining with 50x less compute, ~$0.5M — AccBalanced · 2026-09-09
- Dwarkesh on MagicAILabs' 50x compute claim: RSI may be less compute-bottlenecked than we think — AccBalanced · 2026-09-09
- Rumor: OpenAI may have cracked Navier-Stokes, with human mathematicians laying years of groundwork — LucaAmb · 2026-09-09
- '50% of open problems just solved' — commentator marvels at frontier model progress — BorisMPower · 2026-09-09
- Cheap models via OpenRouter fall apart in agentic harnesses: GLM and DeepSeek can't match Claude — scottyLogJobs · 2026-09-09
- COLM 2026 paper: recent claims that LLMs can introspect don't meet the evidentiary bar — tallinzen · 2026-09-09