Small models inflate benchmark scores by burning reasoning tokens, analysis shows
triclavian · reddit · 2026-11-19
A Reddit analysis introduces the notion of "small model smell": small models can punch above their weight on benchmarks by burning lots of reasoning tokens, but quality collapses sharply when thinking is dialed down — or when a subscription throttles it behind the scenes.
Measuring each model by the ratio of its highest-effort AA score to its lowest, Astra, Fable and Carbon behave most like big models, while Luna shows wild variation — though its Low setting uses far fewer tokens than Haiku's. The takeaway: don't judge small models by leaderboard scores alone; check their stability at low reasoning effort.
More from Models
- Nearly two months post-launch, Opus 5.5 on high reasoning is still the best model — kimmonismus · 2026-11-19
- Gemini 4 Neon hits #2 on Agent Arena as DeepMind's release cadence shrinks to 6 days — lmthang · 2026-11-19
- Theo ranked GPT-5.2 vs Opus 4.5 last year — now he's obsessed with Neon — vivnat · 2026-11-19
- Google's halving release cadence would exhaust all 118 element names by Nov 24 — emollick · 2026-11-19
- Users find gentle workarounds for Fable 5's overfiring safety classifier — repligate · 2026-11-19
- "GPT 6.2 Sol is just pathetic": Burkov shares blunt model-failure screenshot — burkov · 2026-11-19