HF engineer: switching to open models without tuning your eval harness leaves huge capability on the table
VictorSanh · x · 2026-11-05
Victor Sanh of Hugging Face warns that most teams switching from closed to open models keep the old eval harness, judging new models at their worst. A score-vs-cost chart shows models like GPT-6 Luna, Claude Haiku 5.5, GLM 5.3 Flash, DeepSeek V4.1 Flash, Kimi K3 and Gemini 3.8 Flash jumping to near-100% on held-out scenarios after 'Beaker' harness optimization, at $0.01–$0.40 per scenario versus a $0.60 GPT-6 Astra shared baseline. Takeaway: similar performance at an order of magnitude cheaper — if you tune the harness for the new model.
More from Models
- Artificial Analysis details benchmark methodology: every 6 hours, sustained load, agentic sessions — ArtificialAnlys · 2026-11-05
- GLM-5.3's Fastest-Streaming Endpoints Take 18-26s to First Token, Slipping in New Benchmark — ArtificialAnlys · 2026-11-05
- GLM-5.3 endpoint posts 92.8% error rate as Artificial Analysis starts counting failed requests — ArtificialAnlys · 2026-11-05
- Artificial Analysis makes Time per Task its headline metric; default effort cuts task time 3-4x — ArtificialAnlys · 2026-11-05
- Artificial Analysis Overhauls API Benchmarks: Time per Task Replaces Streaming Speed — ArtificialAnlys · 2026-11-05
- Moonshot releases open-weight Kimi K3.1 Pro: 2.8T params, 104B active, 1M context — MaziyarPanahi · 2026-11-05