Agent benchmark scores hinge on container configs: 6-point swing from resource limits alone
krishnan · x · 2026-11-05
krishnan points out that an agent's benchmark score can depend on when its container gets killed, citing Anthropic's February engineering study: same Claude model, same harness, same Terminal-Bench 2.0 tasks—only the resource configuration changed, and success moved 6 percentage points (p < 0.01).
- Mechanism: containers get a guaranteed allocation plus a hard kill limit; set both to the same value and a momentary memory spike kills a run that might have succeeded. At strict settings, 5.8% of tasks hit infra errors.
- 1x→3x headroom: infra errors fell to 2.1%; success moved within noise (p = 0.40). The eval got more stable, not easier.
- Beyond 3x: extra resources let agents pull heavy dependencies, spawn expensive subprocesses, and run memory-hungry test suites, cutting infra errors another 1.6 points.
Takeaway: resource configuration is a hidden variable in agent benchmarks—check the container setup before comparing scores. The attached image is the author's editorial cartoon "Same model, bigger box."
More from coding & agent
- Claude Code cheat sheet: CLAUDE.md, Skills, MCP and the plan-implement-iterate loop — nikola_mr64990 · 2026-11-05
- Bobby Iliev's open-source 'Introduction to Docker' book joins ChapterPal with an AI tutor — burkov · 2026-11-05
- CloudOpsBench: open-source benchmark tests AI agents on 45 real cloud ops scenarios — DarkMG1 · 2026-11-05
- Mid-steps of an AI level-building pipeline: playtest first, then batch concept art with GPT Image — FinanceYF5 · 2026-11-05
- Building a Complete Original Game in 24 Hours with Opus, Flux, Tripo and Three.js — FinanceYF5 · 2026-11-05
- Team automates 100% of product design with Claude, hasn't touched Figma in months — manosaie · 2026-11-05