"Never trust a benchmark you haven't cheated yourself": dev shows agent benchmarks favor whoever builds them
cnakazawa · x · 2026-10-29
Developer boshenc describes a phenomenon that destroyed his trust in benchmarks: if he works on X and you work on Y, X wins when he creates the benchmark and Y wins when you do — each side designs benchmarks around its own implementation.
Concrete example: tsrs's first benchmark made TypeScript look absurdly slow by adding --extendedDiagnostics, a flag not even present in tsrs itself.
cnakazawa's takeaway: "Never trust a benchmark you haven't cheated yourself" — only those who've gamed a benchmark truly understand how it can be manipulated.
More from coding & agent
- Nous Research's New Plugin Auto-Routes Each Subagent to the Right Model — Teknium · 2026-10-29
- Researcher: purely on-policy RL is too sample-inefficient for long-horizon agents — JoshPurtell · 2026-10-29
- Arena skill makes 100 Claude Code sub-agents fight in a tournament for the best answer — blaizedsouza · 2026-10-29
- Why git worktrees remain unavoidable for running parallel coding agents locally — dotey · 2026-10-29
- Durable execution meets agents: decades-old infrastructure assumptions break down — blaizedsouza · 2026-10-29
- Dev Shows Memory Graph of AI TTRPG Engine: 1947 Nodes Growing Each Turn — Edenisb · 2026-10-29