LLM routing saved 33.2% vs premium models in 640-request pilot, but a fixed mid-priced model beat it
smakosh · reddit · 2026-09-26
LLM Gateway self-benchmarked its Jev-based routing against fixed-model baselines across 640 measured requests (80 prompts, four arms, two output limits). Routing cost 33.2% less than a premium baseline but passed 75/79 tasks vs 79/79 after adjusted scoring, and a fixed mid-priced model cost 72.9% less than routing while passing 76/79. The cost confidence interval was wide (4.3% higher to 66.1% lower), and the pilot excluded long conversations, tool use, and agentic coding. Takeaway: routing trims premium bills, but a cheaper fixed model remains a strong alternative. Full prompts, costs, and reproducible scripts are published.
More from Infra
- Running Qwen2.5-Coder-3B on Android: Jupyter + Pydantic Evals on-device eval pipeline — alex_b3b · 2026-11-11
- 22% of Nvidia revenue now books to Singapore, but only ~1% is local buyers — Gregory_C_Allen · 2026-11-11
- Huawei's Kirin 9050 NPU claims offline 30B MoE on smartphones, +140% perf — AIFlow_ML · 2026-11-11
- Stratechery: Intel vs AMD's diverging earnings and Intel's need for an old-new path — Stratechery · 2026-11-11
- a16z Podcast: Mike Volpi on What the 1990s Internet Boom Teaches Us About the AI Buildout — a16z Podcast · 2026-11-11
- Alibaba files to list chip-design unit T-Head in Hong Kong to fund AI push — pstAsiatech · 2026-11-11