Reasoning-Trace Analysis paper: auditing LLM-generated Q-matrices via CoT traces

alex_verem · x · 2026-11-21

The source post linking to the arXiv paper "Reasoning-Trace Analysis Using Large Language Models for Interpretable Q-Matrix Construction" (arXiv:2611.26422, Jihong Zhang, Zijing Jin, Mingfeng Xue).

The paper proposes the RTA framework: an LLM generates attribute-labeled reasoning steps via chain-of-thought prompting, and these traces are analyzed to reconstruct, compare, and inspect each item-attribute assignment — meant to supplement expert review and empirical validation. In simulation and empirical studies (11 fraction-subtraction items, 536 examinees, 4 LLMs, up to 3 temperatures, 100 replications), CoT-derived Q-matrices overlapped substantially with the reference (mean 0.809) but less than directly generated ones (0.877), and varied more across runs. Trace inspection tied disagreements to item-type coding, labeling rules, and how operations were expressed as steps.

Related event: Step-by-step prompting makes LLMs worse at skill tagging, study finds(2 posts)→

Original post →

More from Research

Research channel →