Anthropic: Infrastructure Config Alone Swings Agentic Coding Benchmarks by 6 Points
giansegato · x · 2026-09-29
An Anthropic engineering post shows that infrastructure configuration alone can significantly perturb agentic coding eval results—sometimes exceeding the leaderboard gap between top models.
- In internal experiments, the gap between the most- and least-resourced setups on Terminal-Bench 2.0 was 6 percentage points (p < 0.01), while top leaderboard spots are often separated by just a few points.
- Unlike static benchmarks, agentic evals give models a full environment to write code, run tests, install dependencies, and iterate—the runtime becomes an integral part of problem-solving, so two agents with different resource budgets and time limits aren't taking the same test.
- Terminal-Bench 2.0 now specifies recommended CPU/RAM per task, but specifying resources isn't the same as enforcing them consistently, and enforcement methodology itself can change what the benchmark actually measures.
- The finding emerged when Anthropic's GKE-cluster scores didn't match the official leaderboard during calibration.
The poster adds a caveat: industry benchmarks carry many hard-to-control confounders—don't overindex on them; just try the model.
More from coding & agent
- Agents Now Reply to Slack Messages on Their Own, Resolving a Prod Incident Ghost-to-Ghost — iamrobotbear · 2026-11-16
- Engineer's Interview Prep Method: Use Claude to Find Which Answers Undersell Your AI Work — brandon_galang · 2026-11-16
- DeepSeek open-sources its app frontend "chat-demo", already at 3.9k stars — NielsRogge · 2026-11-16
- Agent asks permission before clicking "I'm not a robot" checkbox — Puzzleheaded-King584 · 2026-11-16
- Generic API-catalog MCP server vs one tool per job: a context-bloat experiment — MandeepKaur9407 · 2026-11-16
- Berkeley paper: LLMs answer with stale values even when the update is in context — ombre-ai · 2026-11-16