Beaker: tuning the harness lifts GLM 5.3 Flash from 29% to 82% on AppWorld
VictorSanh · x · 2026-11-05
The Beaker team shows that swapping models without tuning the harness badly underestimates them. On the AppWorld coding-agent benchmark, GLM 5.3 Flash completed just 29% of scenarios untuned — after Beaker autonomously tuned the harness, completion jumped to 82%, nearly twice as fast (138s→76s) at the same $0.03/scenario cost.
- How it works: Beaker reads agent outputs, analyzes traces, formulates hypotheses about needed changes, tests them, and keeps what improves the agent — fully autonomously.
- Key finding: across 8 models tested, tuning made initially weak models competitive, and the two strongest models needed completely different changes.
- Takeaway: compare models only after tuning the harness for each; switching from closed to open models without tuning leaves capability on the table.
Beaker is now in preview.
More from coding & agent
- Consultant replaces $1,300/mo 'AI employee' agencies with a Claude playbook that recovered $167K — aryanXmahajan · 2026-11-05
- Open-source product-engineering skills ship real products with coding agents, no coding — PawelHuryn · 2026-11-05
- Gemini CLI v0.67.0 ships IDE companion container sandbox bridge and a batch of fixes — gemini-cli-robot · 2026-11-05
- Cloudflare maps out x402 payments so AI agents can pay for the content they read — kleffew94 · 2026-11-05
- Running Qwen at 1Q quant: dev shares DeepSeek Harness fix plus Nemotron 3.5 tips — gnukeith · 2026-11-05
- Perplexity launches MotionSites skill so Computer can build WebGL-animated websites for all users — AravSrinivas · 2026-11-05