UIUC Paper: High Accuracy Hides Flawed Reasoning as LLMs Fail Under Rephrased Logic Problems
UIUC-CS · hf · 2026-10-21
A UIUC paper introduces competence-based evaluation of LLM logical reasoning, exposing limits of accuracy-only benchmarks.
- Current benchmarks report single accuracy numbers on selected tasks and can reward other strategies: prior work found GPT-3.5 scored higher on ReClor and LogiQA2 when prompted to avoid logical reasoning.
- Models that appear to reason well show systematic drops under controlled perturbations, and reasoning models can collapse entirely.
- Competence-based evaluation reveals LLMs fail to consistently apply reasoning rules across varied problem presentations despite high bounded-task accuracy.
More from Research
- Aviation tracks every part's provenance — agents can reason over that graph — colinmcnamara · 2026-10-21
- SFT details: instruction/reasoning/code/math mix, synthetic + human traces, no epoching — TimDarcet · 2026-10-21
- Mid-training recipe disclosed: 32k→131k context extension, dropping logit distillation for tokens — TimDarcet · 2026-10-21
- Mathematician Daniel Litt concedes his AI bet, but says AI won't solve math — ziv_ravid · 2026-10-21
- AI drug discovery reality check: 'give me everything that binds,' then filter for toxicity — burny_tech · 2026-10-21
- PyTorch details KTIR and Dataflow Scheduler behind IBM's Spyre accelerator backend — PyTorch · 2026-10-21