Debate: Should Agent Benchmarks Measure Tokens and Turns or Just Cost and Time

A community debate questions whether token and turn counts are useful agent benchmarks. xeophon argues only cost and wall clock time matter, while critics counter that wall clock unfairly penalizes deployments with weaker infrastructure.

2026-09-17 ~ 2026-09-17 · 3 related posts

• Evals shouldn't be skewed by infra: why turn limits break on sub-agents — langstonnashold · 2026-09-17
• Agent eval harnesses: why one researcher favors 8h wall-clock limits over turn caps — xeophon · 2026-09-17
• Tokens and turns are useless eval metrics — only money and wall clock matter — xeophon · 2026-09-17