Wharton Report: Chain-of-Thought Prompting Shows Diminishing Returns in Modern Models
emollick · x · 2026-08-02
A new technical report from Wharton Generative AI Labs reveals that the classic "think step by step" (Chain-of-Thought) prompting is losing effectiveness on modern AI models.
- Non-reasoning models: Show modest average improvements but significantly increased answer variability.
- Reasoning models: Gain only marginal benefits despite a substantial 20-80% increase in response time.
By testing each question 25 times per condition, the research exposes inconsistencies masked by traditional one-time tests, challenging the assumption that CoT is universally beneficial.
More from Research
- Simplex Diffusion Models paper talk by EPFL/DeepMind's Justin Deschenaux on Oct 19 — ArnaudDoucet1 · 2026-10-16
- Meituan LongCat team's Data World synthesizes data environments to train data agents — meituan-longcat · 2026-10-16
- FDE-Gym benchmark shows huge gap on real AI delivery tasks: top model scores 0.563 — DeepLethe · 2026-10-16
- ForcingBooster halves computation of autoregressive video diffusion with patch-wise denoising — VAST-AI · 2026-10-16
- Baidu open-sources FlashMask: up to 1.55x faster than FlashAttention-4 for long-context training — BaiduResearch · 2026-10-16
- SemEval-2027 Task 9 opens: multimodal news framing analysis across 4 languages — preslav_nakov · 2026-10-16