Rerunning LLM eval 290 times shows single-pass comparisons are just luck
ActiveStriking2719 · reddit · 2026-08-31
Author previously compared Haiku 4.5 and Sonnet 5, finding the expensive model won by just one question. Prompted by a reader, he reran the 29 questions 5 times (290 total answers), revealing significant changes:
- Stability: Both models had zero fatal errors, but in some runs, the cheaper model scored higher. The original "deciding" question (bottle cap matching) flipped completely in subsequent runs.
- Key Finding: A single-pass eval is just one sample from a distribution and can "crown" an outlier due to randomness. The original conclusion "pick the cheap one" stands, but based on stability and cost, not a one-question margin.
- Limitation: Reruns prove the test's stability but not generalization to unseen questions.
Takeaway: Run small-sample evals at least 5 times before publishing margins to avoid being misled by randomness.
More from coding & agent
- True async delegation hides the complexity of thinking traces and tool calls — manosaie · 2026-08-31
- Hermes Agent: the free open harness that turns local LLMs into working agents — Teknium · 2026-08-31
- Teknium: Hermes + Claude Outperforms Claude Code in Benchmarks — Teknium · 2026-08-31
- Same Model, Different Harness: Coding-agent results vary by context policy — rohanpaul_ai · 2026-08-31
- Best approaches for integrating AI agents into existing products? — SellingN8 · 2026-08-31
- Request for help: Building an MCP server for a SaaS app from scratch — WriterNorth2299 · 2026-08-31