LLM chain-of-thought hurts Q-matrix accuracy: 80.9% vs 87.7% expert match, study finds

alex_verem · x · 2026-11-21

Researchers at the University of Arkansas, Pitt, and UNC Greensboro had 4 LLMs label the math skills tested by 11 fraction-subtraction items (Q-matrix construction), running every setup 100 times for 2,000 skill maps, then compared against expert maps and real responses from 536 students.

Key findings:

The authors recommend running models many times, summarizing results, and keeping a human expert in the loop. Paper: arXiv:2611.26422 (Reasoning-Trace Analysis framework).

Related event: Step-by-step prompting makes LLMs worse at skill tagging, study finds(2 posts)→

Original post →

More from Models

Models channel →