pass@k vs pass^k: an 80% agent passes 4 straight runs only 41% of the time
cwolferesearch · x · 2026-11-20
Cameron Wolfe explains why agent evaluation must measure both capability and reliability, focusing on the pass^k metric from τ-bench.
- pass@k: probability of at least one success in k attempts — rewards occasional success
- pass^k: probability of succeeding in all k independent attempts — rewards consistency
- The same agent with an 80% success rate scores Pass@4 = 99.84% (near-perfect) but Pass^4 = 40.96% and Pass^8 = 16.78% (unreliable)
Reliability is harder for agents because long sequences of tool calls, reasoning steps, and environment interactions each add failure opportunities with small variations.
When to use which: a coding agent succeeding after retries is fine (pass@k); an agent autonomously modifying production systems needs pass^k. The attached image includes the formulas and computation code for both metrics.
More from coding & agent
- DeepMind's Antigravity Robotics Plugin: 2 Prompts, ~20 Minutes to Command a Real FR3 Duo Arm — jackyliang42 · 2026-11-21
- Ruff author warns: setting MYPY speeds up type checking on Pydantic but collapses types to Mapping[str, Any] — charliermarsh · 2026-11-21
- Asking Astra 6.2 to Review Code Written by Opus 5.5 Yields Almost Nothing — rickasaurus · 2026-11-21
- Security & Web3 Firewall MCP Server Hits Official Registry: Detects Tool-Poisoning and Rug-Pull Drift — ezelech · 2026-11-21
- Artificial Analysis Launches Search API Benchmark Covering 26 Products Across 14 Providers — ArtificialAnlys · 2026-11-21
- Pydantic hacks Mypy detection: why type checkers crawl and how to fake your way to speed — charliermarsh · 2026-11-21