pass@k vs pass^k: an 80% agent passes 4 straight runs only 41% of the time

cwolferesearch · x · 2026-11-20

Cameron Wolfe explains why agent evaluation must measure both capability and reliability, focusing on the pass^k metric from τ-bench.

Reliability is harder for agents because long sequences of tool calls, reasoning steps, and environment interactions each add failure opportunities with small variations.

When to use which: a coding agent succeeding after retries is fine (pass@k); an agent autonomously modifying production systems needs pass^k. The attached image includes the formulas and computation code for both metrics.

Original post →

More from coding & agent

coding & agent channel →