Reasoning-Trace Analysis paper: auditing LLM-generated Q-matrices via CoT traces
alex_verem · x · 2026-11-21
The source post linking to the arXiv paper "Reasoning-Trace Analysis Using Large Language Models for Interpretable Q-Matrix Construction" (arXiv:2611.26422, Jihong Zhang, Zijing Jin, Mingfeng Xue).
The paper proposes the RTA framework: an LLM generates attribute-labeled reasoning steps via chain-of-thought prompting, and these traces are analyzed to reconstruct, compare, and inspect each item-attribute assignment — meant to supplement expert review and empirical validation. In simulation and empirical studies (11 fraction-subtraction items, 536 examinees, 4 LLMs, up to 3 temperatures, 100 replications), CoT-derived Q-matrices overlapped substantially with the reference (mean 0.809) but less than directly generated ones (0.877), and varied more across runs. Trace inspection tied disagreements to item-type coding, labeling rules, and how operations were expressed as steps.
Related event: Step-by-step prompting makes LLMs worse at skill tagging, study finds(2 posts)→
More from Research
- Claude's claimed Riemann proof mocked: 250,000 pages of unreviewed Lean, author calls it 'slop' — GaryMarcus · 2026-11-21
- GGUF 4-bit quantization is just GIF palette compression, 30 years later — carrigmat · 2026-11-21
- Proposed GRH proof by Alessandro Masullo released early; Brian Conrad vetted it for months — zainhas · 2026-11-21
- Human-proposed GRH proof rushed public amid Claude Riemann hype: 'all human, no AI' — zainhas · 2026-11-21
- CT-Bench, a benchmark for LLM agent cooperation via self-negotiated contracts, accepted at COLM 2026 — maxhkw · 2026-11-21
- Frank Nielsen demos geodesic right-angle triangles in dually flat spaces — FrnkNlsn · 2026-11-21