Benchmarking AI assistants: measure time to a checked result, not time to an answer
OriginalHospital · reddit · 2026-09-10
The author proposes defining "finished" before comparing AI assistants: a response appearing on screen is one event, a usable result another.
A fictional example: assistant A drafts in 1 minute but needs 14 minutes of checking; assistant B takes 4 to generate and 3 to review — A looks faster if the stopwatch stops at generation, but B reaches the defined finish sooner. For source-based summaries, "finished" means every claim verified, caveats kept, output fit for the audience.
Tips for fair small trials: same sources and acceptance criteria, count failed attempts, don't give one assistant extra context or retries, blind model names, and rotate review order since learning the source on the first answer biases the second. Report generation and review time plus residual errors separately — don't turn a tiny comparison into a universal ranking.
More from coding & agent
- Dev asks: hundreds of AI agents share one API key — is per-agent identity worth it? — SheepherderFree3931 · 2026-09-10
- Stripe Built an SDK Prototype in 2 Days Instead of 3 Weeks by Delegating to AI Agents Spec-First — dl_weekly · 2026-09-10
- Veteran dev: agentic coding loops 8 times, burns 100k tokens, and wrecks mature codebases — cgouguen · 2026-09-10
- Replacing smolagents' VisitWebpageTool with a custom fetch layer to beat anti-bot pages — InsideDebt6345 · 2026-09-10
- AI agents don't know their code makes codebases harder to work with — _Stocko_ · 2026-09-10
- Hy4 preview tested: playable 3D survival game from a single prompt in WorkBuddy — mhdfaran · 2026-09-10