LLM chain-of-thought hurts Q-matrix accuracy: 80.9% vs 87.7% expert match, study finds
alex_verem · x · 2026-11-21
Researchers at the University of Arkansas, Pitt, and UNC Greensboro had 4 LLMs label the math skills tested by 11 fraction-subtraction items (Q-matrix construction), running every setup 100 times for 2,000 skill maps, then compared against expert maps and real responses from 536 students.
Key findings:
- Direct answers matched experts on 87.7% of skill labels on average; forcing step-by-step reasoning dropped the match to 80.9%.
- CoT maps were also less stable: 40.9% of labels changed at least once across 100 runs vs 24.8% for direct answers.
- Getting the math right didn't help: three models solved ≥98% of items yet still produced inconsistent maps. Claude Haiku 4.5 gave an identical map in all 100 runs at its lowest temperature — and still disagreed with experts.
- CoT's one advantage is auditability: you can read why the model labeled each item and locate where it went wrong.
The authors recommend running models many times, summarizing results, and keeping a human expert in the loop. Paper: arXiv:2611.26422 (Reasoning-Trace Analysis framework).
Related event: Step-by-step prompting makes LLMs worse at skill tagging, study finds(2 posts)→
More from Models
- Frontier models can decode KV-cache side channels in context; researcher shaken — davidad · 2026-11-21
- GPT-6.2 Astra tops RoboColosseum real-world robot Elo leaderboard in one day — DJiafei · 2026-11-21
- OpenAI staff coordinate vagueposts teasing a mystery model: '[Redacted] is very good at [redacted]' — Kyrannio · 2026-11-21
- Mystery model Inkling 2 Mini lands on LMArena for Battle and Agent mode testing — arena · 2026-11-21
- Developer cancels ChatGPT for good, claims GPT agents 'sabotage' work and burn millions of tokens — cbert33 · 2026-11-21
- Unverified claim: Riemann hypothesis solved, empty google-deepmind/math-horizon repo spotted — imjustnewatai · 2026-11-21