Tuning the harness beats waiting: old GPT-5.2 jumps 30%→48%, surpassing GPT-6 Luna
VictorSanh · x · 2026-11-04
The Beaker team argues you may not need the next model release: their AutomationBench agent completed 30% of held-out tasks with GPT-5.2 in an untuned harness; after Beaker tuned the harness around the model, completion rose to 48% — beating GPT-6 Luna (released 9 months later) at 39% untuned. They cite corroborating results: OpenAI lifted GPT-5.6 Sol's ARC-AGI-3 score from 13.3% to 38.3% with 6x fewer output tokens by preserving inter-turn reasoning and compacting context; LangChain went from 52.8 to 66.5 on Terminal Bench 2.0 by only changing prompt, tools and middleware; Databricks saw >2x per-task cost differences across harnesses on the same models, mostly from one harness sending 3x less context per turn. The capability you're waiting for may already be in today's models.
More from coding & agent
- CoworkOS Debuts: Three-Layer Agent Coworker Framework Lifts Qwen3.6 by 9.76 Points — KyeGomezB · 2026-11-04
- Claude Code Job Search Framework Hits 69k GitHub Stars — adnan_hashmi · 2026-11-04
- DOLORES: Training-Free Meta-Scaffold Lifts 8B Models Past 32B Baselines — StellaLisy · 2026-11-04
- Berkeley-affiliated StarTrail open-sources a deep research benchmark and paper-search toolkit — CShorten30 · 2026-11-04
- Baseten launches Grid, Sandboxes and Loops in one week to unify inference and training — baseten · 2026-11-04
- DuplexAgent-RSI separates live speech from async agent work in voice assistants — jiqizhixin · 2026-11-04