Do Benchmarks Really Reflect Real-World Workflows? Four Objections

Lucky_Creme_5208 · reddit · 2026-10-24

After reviewing Artificial Analysis's evaluation methodology, the author raises four objections:

Over-restriction: many evals ban internet access, disable tools, and cap steps and time, unlike real workflows.
Subjective grading: whether LLM judges are reliable is doubtful, and human grading varies by taste and quality bar.
Custom harnesses: evaluators build their own harness (e.g. stirrup) that nobody uses in practice.
Public datasets: open data invites benchmaxxing during training.

He argues the ideal benchmark uses private data, objectively defined grading, no unnecessary restrictions, and a widely used harness.

More from Models