Why Surgery Benchmarks Reward Fine-tuned Small Models While Math Benchmarks Don't

ddonoho · x · 2026-09-15

The author shares a non-obvious insight from their surgery benchmark: specialized-skill benchmarks don't all behave like math benchmarks.

On math benchmarks (e.g., FrontierMath), frontier models dominate in descending order of size/release date, and fine-tuned mini-models are non-players.
On the surgery benchmark, however, fine-tuned small models (like LemonFM) top the leaderboard — and the benchmarks were created by independent teams, so there's no self-grading bias.
The proposed explanation: essentially all of math is "publishable" and thus embedded in pretraining data, while surgical practice knowledge is held by a small group and never systematically digitized, so general models can't learn it and fine-tuning becomes the only path.

Implication: the more封闭 the domain knowledge, the larger the advantage of fine-tuned small models over general frontier models.

More from Models