Reasoning-Trace Analysis Using Large Language Models for Interpretable Q-Matrix Construction
Jihong Zhang, Zijing Jin, Mingfeng Xue
cs.AI
2026-11-20
Skill-labeled CoT traces rebuild the Q-matrix for diagnostic tests: .809 overlap with the expert reference versus .877 for direct generation, but every entry is now traceable.
Diagnostic classification models (DCMs) sort examinees into discrete skill-mastery profiles, for instance masters simplifying fractions but not borrowing. Everything rests on a Q-matrix, a 0/1 table marking which attributes each item requires, traditionally built by expert judgment: slow, subjective, and error-prone, with a wrong cell distorting item parameters and examinee classifications alike. Recent work shows LLMs can draft Q-matrices that approximate expert versions, but evaluation stops at the finished matrix. Whether a given 1 reflects the model deciding the item needs borrowing, or recall of a matrix published in 1990, cannot be told apart. For a field built on validity evidence, an unauditable judgment is a compliance problem.
RTA (Reasoning-Trace Analysis) has one core move: the LLM states, for each solution step, what it does and which skill it exercises, and the matrix is rebuilt from those labels so the basis for every assignment lands on paper.
The idea borrows the explicit rule-following principle of RFFT but applies it at inference time, with no fine-tuning. The baseline is a direct workflow: same attribute labels, one call, full 11×4 matrix, no intermediate steps.
The simulation study first checks whether traces carry signal. All 10 perturbed items require all four attributes; DeepSeek v4 Flash, Claude Haiku 4.5, and Kimi K2.6 answered every one correctly and recovered every attribute. GPT-4o mini answered only half correctly yet recovered the full four-attribute vector on 8 of 10. On six variants engineered to drop an attribute, all four models withheld it, with consensus overlap of .958 to 1.000 against a .625 baseline for blindly marking everything 1. The labels carry real signal.
The empirical study used 11 fraction-subtraction items, responses from 536 examinees, 20 conditions × 100 replications, 2,000 generated matrices:
| Metric | CoT workflow | Direct workflow |
| Mean overlap with reference | .809 | .877 |
| Exact item match (all 4 attributes) | .364 | .564 |
| Unstable cell rate U | .409 | .248 |
| Dominant-matrix share D | .044 | .540 |
Interpretability has a price: lower alignment and less stability. Temperature 0 does not suppress it; DeepSeek's unstable-cell rate at temp 0 was still .591. Across the four CoT conditions, disagreements ran 29 omissions against 3 over-labelings, and every consensus matrix came out sparser than the reference.
Fit adds a second lens. The DeepSeek CoT matrix overlapped only .818 with the reference yet fit examinee responses slightly better under DINA (−2LL 5044.9 vs 5045.8), while the GPT-4o mini direct matrix overlapped .795 and fit 347 points worse. Across the 13 distinct matrices, the rank correlation between overlap and fit was −.49 (p = .09), so neither index vouches for the other. The reference itself is not clean: PVAF and Hull flagged 3 of its cells for changing from 1 to 0, and the two cells LLMs departed from most often are on that list. Under G-DINA, 64% of examinees received the same profile as under the reference. Of the 2,000 generated matrices, the 20 consensus matrices, and the reference, not one meets DINA's structural identifiability conditions.
The selling point is auditability, not accuracy. A direct workflow offers no process to inspect, and its higher alignment could come from memorizing a public item set; a follow-up on isomorphic items with changed numbers kept alignment largely intact, which mostly rules out pure retrieval. Under responsible-AI requirements for measurement, an error you can point to in a specific step beats a match nobody can verify.
Two findings transfer beyond testing. A correct answer does not mean a correct label: GPT-4o mini got half the arithmetic wrong while attributes stayed recovered, and three near-perfect models still produced wildly varying matrices. Agreement with a reference is not validity either: the empirical validation methods disagreed with the reference on the same cells the LLMs did.
The authors list six. The three that matter most: everything comes from a single fraction-subtraction item set (4 attributes, 11 items, far from a real pool); three of the four models are lightweight variants; the two workflows differ in call structure (11 item-level calls vs one full-matrix response), which may explain part of the stability gap. Traces are generated text, not the model's internal computation, and the skill labels were never independently coded.
Two things remain unattributed: how much of the .068 alignment drop comes from the reconstruction rule folding operations into neighboring labels versus genuine failures to recognize an attribute (the supplementary experiments separate only part of it), and why the unstable-cell rate reaches .591 at temperature 0.