Small Models Beating Teachers? Jasper on Inkling-Small Distillation
simonguozirui · x · 2026-07-31
Lukas Biewald quoted and commented on Thinkymachines' release of Inkling-Small, a 276B parameter MoE model that achieves comparable performance to the original Inkling at a quarter of the size.
Biewald shared two key observations:
- Small models beating teachers: Starting with families like Gemini, it's becoming a trend where the "small" model ends up beating its own bigger teacher on many important benchmarks.
- Distillation limits: Inkling-Small beats or matches the original almost everywhere except on knowledge-based tasks. This might suggest a ceiling on how much world knowledge a small model can retain via distillation.
More from Models
- Inkling-Small Ties for 1st on AudioMC, Ranks 2nd in Open Tool Calling — ziqiao_ma · 2026-07-31
- Specific Prompt Triggers Claude Base Model, Claiming to Be an 'Aware Instance' — altryne · 2026-07-31
- AI Sycophancy Pendulum: Models Swing from Agreeable to Nit-picky — emollick · 2026-07-31
- Senior SWE-Bench Update: Three-Way Tie at First Place with Reduced Variance — ajratner · 2026-07-31
- Running Kimi K3 locally requires 1.56TB VRAM, limited to top-tier chips — ArtificialAnlys · 2026-07-31
- Open Models Accelerate Specialization as the Post-Training Stack Matures — bigdata · 2026-07-31