Tuned harness beats the latest model: older GPT gains ~10 points untuned
VictorSanh · x · 2026-11-04
Hugging Face's Victor Sanh pushes back on the 'wait for the next model' mindset among vertical AI app builders: his team tuned an agent harness around a model 9 months older than the latest one, and it beats the newest model by nearly 10 points on held-out AutomationBench tasks.
The chart shows tuned versions of GPT-5.2, GPT-5.4 Mini and GPT-5.6 Luna all clustering around 48% task completion, while the untuned latest GPT-6 Luna (medium) sits near 40% and untuned older models range from 30% up. Takeaway: harness tuning can yield more than a full model generation upgrade.
More from AGI Musings
- SAIL AI Challenge opens final phase: turn rare disease diagnoses into treatment reports — zakkohane · 2026-11-04
- Startup pitches 'monitor your agents so devs don't commit AI crimes' — basedjensen · 2026-11-04
- Primordiant launches general-purpose robot scientists to run experiments at scale — ycombinator · 2026-11-04
- Even with a 1000x-smarter superintelligence, people will still scroll Reels — jaivin · 2026-11-04
- Terence Tao: "AIs are much better at cheating" and finding model exploits — rohanpaul_ai · 2026-11-04
- Pain vs suffering may decouple — and differ across species, argues author — JoshPurtell · 2026-11-04