More thinking, worse scores: refusals tank closed-model evals at max reasoning effort
_lewtun · x · 2026-10-14
Lewis Tunstall (Hugging Face) highlights a quirk when evaluating closed models on tasks that trigger safety refusals (cyber, bio, etc.):
- Low reasoning effort → few refusals, high accuracy
- Max reasoning effort → many refusals, low accuracy
This inverts test-time compute curves: max effort performs far worse than low effort because refusals crater the scores. He asks what the fairest way to report performance is in these cases.
More from Models
- OpenAI status page confirms degraded performance for ChatGPT, Codex outage ongoing 55+ minutes — bdsqlsz · 2026-10-14
- Gemini 4 burns through Google AI Pro rate limits in 10-20 messages, user reports — NumLock2022 · 2026-10-14
- Investor: Argon Is a Big Step Up but Costly to Serve and Still Behind on Coding Agents — firstadopter · 2026-10-14
- YouTuber goes hands-on with invite-only Gemini 4 Argon, Google's next frontier model — awsconsultant · 2026-10-14
- Are these benchmarks real? Reddit questions Gemini 3.8 Flash's standout scores — Numerous-Campaign844 · 2026-10-14
- ChatGPT voice mode appears to rewrite user prompts mid-conversation — Appropriate_Case2976 · 2026-10-14