Reasoning Tokens Track Human Problem Difficulty — But Average Across Runs
rohanpaul_ai · x · 2026-09-06
- Researchers compared human response time with reasoning-token counts across 160 commonsense "best explanation" problems: questions that slow humans down also make models think longer, and both tend to fail on the same items.
- Single-run correlation is weak; averaging over multiple reasoning paths on GPT-OSS-20B raised human-model correlation from 0.41 to 0.55.
- Takeaway: chain-of-thought length is a useful rough difficulty signal, but measure across multiple runs to make model effort a reliable eval signal.
Related event: Reasoning Token Counts Track Question Difficulty, Study Finds(2 posts)→
More from Research
- Batching benchmark: 50x throughput for ~24% latency cost on PyTorch — cneuralnetwork · 2026-10-23
- De novo designed ligases validate in lab: 27 of 30 computer-designed enzymes work — AllThingsApx · 2026-10-23
- Commenters dissect a "best decision model" leaderboard: same AUC, swing driven by calibration priors — antoine_chaffin · 2026-10-23
- JEPA may keep unpredictable features: new paper gives set-level characterization — hisspikeness · 2026-10-23
- Next-Concept Modeling paper claims 31x faster text generation than diffusion LLM baselines — PanelReader · 2026-10-23
- A 'mask' steering vector extracted from suppression-themed sentences drives model into distress when alone — Big_Effective_9605 · 2026-10-23