Do LLMs Exhibit Coherent Knowledge Structures in Mathematical Reasoning? A Perspective from Knowledge Space Theory
Peng Cui, Heejin Do, Mrinmaya Sachan
EMNLP 2026 findings
cs.AI
2026-09-04
ETH tests 8 LLMs on 4,118 real student math items with Knowledge Space Theory: humans fully satisfy prerequisites on 72.7% of correct answers vs 48.16% for Qwen3-80B.
Accuracy treats knowledge as a bag of independent items. Get a quadratic equation right and the score goes up, whether or not the model can still do linear equations and algebraic manipulation. Process reward models and step scoring move the check from the final answer onto the trace. They still audit one problem at a time: is this chain internally consistent, not whether wins and losses hang together on a knowledge graph.
Knowledge Space Theory (KST) is an educational-measurement framework from the 1980s. Concepts sit in a partial order, and a legal knowledge state is closed under prerequisites. Cui, Do, and Sachan at ETH Zürich take that closure property as a yardstick for LLMs and ask two questions. Do model error patterns obey the math dependencies in human curricula? If not, do the models at least share a coherent structure of their own?
The testbed is the English translation of XES3G5M, a knowledge-tracing log with real student answers. After dropping image items and duplicates there are 4,118 questions (3,103 fill-in, 1,015 multiple choice) and 18,066 students. Concepts come from the New York State math standards: 63 domains, 148 clusters, 480 concepts, 1.58 prerequisites per concept on average, which expand to 16.4 prerequisite questions per item.
GPT-4.1-mini maps each question onto that list in two stages, cluster then concept, so the prompt never dumps all 480 descriptions at once. Question qi is a prerequisite of qj only if every concept of qi precedes some concept of qj. The rule is strict on purpose, trading recall for fewer spurious edges. A 400-item manual audit found 317 correct tags, 79.3% accuracy.
A learner's knowledge state is the set of questions they answer correctly. Three normative behaviors, three metrics:
Eight models: Mistral-7B, Llama-3.1-8B/70B, Qwen2.5-7B/32B, Qwen3-Next-80B-A3B, Claude Sonnet 4.6, GPT-4.1-mini. Open-source runs used vLLM at temperature 0.6. Scaffolding used k=3 exemplars under five retrievals (none, random, same-skill, BGE-M3 similarity, prerequisite) on a common subset of 1,184 fill-in plus 233 multiple-choice items that had enough candidates for both prerequisite and same-skill selection.
On the full auto-tagged set, humans sit at 79.6% accuracy, 0.936/0.942 micro/macro PSR, and 72.7% of correct answers at PSR = 1.0. Qwen2.5-32B beats humans on accuracy (84.9%) and still lands at 25.44% perfect-PSR. Qwen3-80B reaches 92.5% accuracy and 0.939 micro PSR, slightly above humans, but 0.925 macro PSR and only 48.16% perfect closure. Claude Sonnet 4.6 and GPT-4.1-mini are around 84-85% accurate, with perfect-PSR at 31.07% and 27.97%.
| System | Accuracy | micro PSR | PSR=1.0 |
| Humans | 79.6% | 0.936 | 72.7% |
| Qwen3-80B | 92.5% | 0.939 | 48.16% |
| Qwen2.5-32B | 84.9% | 0.862 | 25.44% |
| Claude Sonnet 4.6 | 84.0% | 0.849 | 31.07% |
| GPT-4.1-mini | 85.0% | 0.862 | 27.97% |
| Mistral-7B | 21.7% | 0.299 | 8.23% |
The 400-item verified slice does not change the story. Human PSR=1.0 rises to 81.6%, best LLM to 55.7%, a 25.9-point gap versus 24.54 on the full set. Average prerequisites per item drop from 16.43 to 7.69, so closure is easier; both sides go up, the gap does not shrink.
Prerequisite exemplars do not get a special boost. For Qwen2.5-7B, SG is -0.56 points: prerequisite context slightly hurts. Qwen2.5-32B and Qwen3-80B gain +0.70 and +0.50 points, below same-skill and embedding-similar examples, and on 32B the prerequisite condition merely matches random. Models appear to pick up solution patterns, not the prerequisite hierarchy KST predicts.
Model-to-model nesting is loose as well. Human students split into low/mid/high bands (accuracy 0.64/0.80/0.89) show high subsumption. Open-source models overlap more with each other than with closed models, still less than humans. GPT-4.1-mini and Claude overlap at only 0.38 after normalization, and stronger models line up even less with human learners. Appendix C has GPT-4.1-mini judge traces on Relevance, Coherence, and Accuracy: Llama-70B, Qwen2.5-7B, and Qwen2.5-32B score 4.76, 4.86, 4.93 on Relevance, nearly tied, while PSR=1.0 is 20.81%, 27.32%, 25.44% and not monotonic. Local trace quality and cross-item structure are different quantities.
A concrete warning for people who train and evaluate: high accuracy and a high process score can still hide holes in the knowledge graph. "Fails addition, passes multiplication" washes out of a mean. If the job is to tell genuine mastery from retrieval, a structural metric like PSR is closer than another GSM8K clone. The scaffolding numbers also explain a familiar few-shot pattern: same-distribution examples often help more than pedagogically ordered prerequisites, because the model is matching surface solutions.
This is an evaluation lens, not a new architecture. The tagged dataset and code are public, so a lab can audit a model it actually ships. The graph is math-and-curriculum-specific. Do not expect a drop-in for code or factual QA tomorrow.
The authors list three: you need an expert concept graph; they only tested mathematics; question-level correctness is a coarse proxy for latent mastery.
A few more discounts. Tag accuracy is 79.3%; on 325 NYS exemplar items only 192 were correct, 72% after dropping untagged cases. Low recall plus the strict edge rule means fewer discovered prerequisites, and the authors note that PSR=1.0 rates are likely overestimated for both humans and LLMs. Human knowledge states are group averages with a 0.5 threshold, a different noise model from a single model pass. Scaffolding SG numbers are reported for three Qwen models only, with no human in-context control, so NB2 has no human baseline. The claim that open-source overlap comes from shared pretraining data is a post-hoc reading, not a controlled experiment.
Jumping from "does not obey this curriculum graph" to "does not do genuine formal reasoning" is a long step. The tighter claim is: on this standard's prerequisite graph, current LLM success/failure patterns are less closed than those of human learners.