GLM 5.3 Flash jumps 29% to 82% on AppWorld after Beaker tunes the agent harness
VictorSanh · x · 2026-11-17
Beaker's team argues that agent harnesses and prompts are usually built for one model, so swapping models unfairly penalizes them. On the AppWorld coding-agent benchmark, GLM 5.3 Flash completed just 29% of scenarios untuned — but after Beaker autonomously tuned the agent for it, it scored 82% at nearly identical cost ($0.029→$0.033 per scenario) and almost 2x speed (138s→76s). Across eight models, Beaker's per-model optimization turned initially weak models competitive, showing the two strongest models needed completely different adjustments and reframing model choice as a tradeoff between quality, cost, and latency rather than leaderboard position.
Related event: Beaker: Tune Your Agent Harness Before Comparing Models(2 posts)→
More from coding & agent
- Together AI's agent eval workflow lifted task completion from 67% to 96% — togethercompute · 2026-11-17
- Together AI shares its agent eval playbook, lifting task completion from 67% to 96% — togethercompute · 2026-11-17
- Beaker tunes your agent harness per model before you pick one — VictorSanh · 2026-11-17
- VapiCon 2026 takeaways: phone calls still dominate voice AI, sims and evals go mainstream — chadwallacehart · 2026-11-17
- YC F26 startup OpenScaffold turns past deployments into reusable playbooks for coding agents — ycombinator · 2026-11-17
- Vault-LD: markdown wiki + YAML-LD knowledge graph as an open standard for agent memory — _AustinCalvert_ · 2026-11-17