Agent passes 35/54 runs but the 95% CI spans 44%–83%: how to report agent evals properly
cneuralnetwork · x · 2026-11-04
@singhrounakkumr highlights a statistical trap in agent benchmarking: an agent passing 35 of 54 runs (65%) carries a 95% confidence interval of 44.4%–83.3%, so single headline pass rates are misleading. He publishes his repolace benchmark, full run logs, and a write-up admitting where his own reporting went wrong, arguing agent evals must report intervals and variance across runs rather than one-shot scores.
More from coding & agent
- Excalidraw ships improved MCP: text fixes, better perf, Mermaid support — Vjeux · 2026-11-04
- Google's Jules coding agent shuts down Nov 11; user pens farewell with an agent-written book — Free-Stage-5975 · 2026-11-04
- DeepSeek Harness creator mode demo prompts published as copy-paste list — EAccelerate_42 · 2026-11-04
- Perplexity's Computer Agent Adds Agent Plugins for Bundled App Skills — AravSrinivas · 2026-11-04
- Vespin Launches as an Open-Source Inbox for Managing All Your Coding Agents — Scobleizer · 2026-11-04
- Dev revives a dusty Meta Portal with a coding agent; tip: let second-hand units 'decant' for days before login works — msg · 2026-11-04