Beaker tunes agent harnesses: GPT-5.2 task completion jumps from 30% to 48%
VictorSanh · x · 2026-11-04
The Beaker team (preview launch) argues the capability you're waiting for may already be in today's models — if you fix the harness.
Key numbers: On their AutomationBench agent, GPT-5.2 fully completes 30% of held-out tasks untuned; after Beaker tunes the harness, 48% — beating a model released nine months later (39%).
Supporting cases:
- OpenAI tripled GPT-5.6 Sol's ARC-AGI-3 score (13.3% → 38.3%) with 6x fewer output tokens by keeping inter-turn reasoning and compacting context
- LangChain went 52.8 → 66.5 on Terminal Bench 2.0 changing only prompt, tools, middleware
- Databricks measured >2x cost-per-task differences across harnesses on the same models, mostly from 3x less context per turn
How Beaker works: reads agent outputs, analyzes traces, formulates hypotheses, tests changes and keeps what improves the agent — tuned per model on 120 training tasks. The harness decides how much of the model you actually get.
More from coding & agent
- Karpathy's viral take: engineers' biggest regret is unsaved AI conversations — bfrench · 2026-11-04
- Apple lets apps bundle Siri skill files, devs mock the sloppy docs — msg · 2026-11-04
- YC-backed Simantic adds a simulation layer so AI agents can test firmware at software speed — ycombinator · 2026-11-04
- Bittensor Subnet 33 Launches Skill Store: Tested Skills Aim to Make Open Models Do Frontier Work — markjeffrey · 2026-11-04
- Building a 24-hour AI newsroom on Microsoft Copilot Harness: workflow beats prompts — WirelessLife · 2026-11-04
- Polyphonic to give AI agents their own Mac via open-source terminal-mac — RileyRalmuto · 2026-11-04