Three runs, two significant losses: your cheap-model switch may just have been luck
Ok-Challenge-7810 · reddit · 2026-09-12
The author benchmarked an incumbent model vs a cheaper one on CRMArena—same 100 tasks, live Salesforce org, exact-match grading, paired, three independent runs. The cheaper model was 16% and 19% worse (both significant), but the third run showed only 8%, with confidence intervals crossing zero. If that single run had been your one test, you'd have shipped the switch.
Key findings
- One run per task gives a mean, nothing about spread: with 3 runs per task, the incumbent gets 76% of tasks right at least once but only 54% every time—a quarter of tasks flip on every attempt, and that variance hides inside every single-run comparison.
- Worse: best-of-3 for the new model vs one run for the incumbent (a common vendor setup) shows the cheaper model +2% better.
- Properly pooled paired results: 16% worse, CI 4–26%. Even with 3× data, 100 tasks can say "worse" but never "within 5%"—not significant ≠ equivalent.
Cheaper data
Every database lookup the agent makes is a true fact: the author turned those into 95 auto-generated tasks (answer from the database, question written by a model that never sees the answer, zero labelling). Four of six task types behave like hand-written ones—your needed sample size is a generator away. Full notebook linked.
More from coding & agent
- Reddit debate: agent memory, not the model, is the real bottleneck — Luvena21 · 2026-09-12
- You're treating Claude like a junior intern: 8 copy-paste prompts for senior-level AI collaboration — ahuja_priyank · 2026-09-12
- Decagon shares 19+ ablations on using GEPA for test-driven prompt optimization in production — kastnerkyle · 2026-09-12
- GeoGuessr as an RL env: 4B VLM trained with OpenEnv and TRL to play the game — SergioPaniego · 2026-09-12
- Chinese firm used Claude Code to run 20+ dating apps with 4,700 AI personas deceiving 25,000 users — luisdans · 2026-09-12
- Security researcher: cloud providers need defensive agents and rogue-GPU kill plans — kuza55 · 2026-09-12