Small models inflate benchmark scores by burning reasoning tokens, analysis shows

triclavian · reddit · 2026-11-19

A Reddit analysis introduces the notion of "small model smell": small models can punch above their weight on benchmarks by burning lots of reasoning tokens, but quality collapses sharply when thinking is dialed down — or when a subscription throttles it behind the scenes.

Measuring each model by the ratio of its highest-effort AA score to its lowest, Astra, Fable and Carbon behave most like big models, while Luna shows wild variation — though its Low setting uses far fewer tokens than Haiku's. The takeaway: don't judge small models by leaderboard scores alone; check their stability at low reasoning effort.

Original post →

More from Models

Models channel →