Model routing cuts LLM errors 46% at same cost, Martian study finds
SucceededMind · x · 2026-09-04
Martian's "AI Frontier" study: routing across models yields 46% fewer errors than the single best LLM across 16 widely-used benchmarks (TerminalBench, LiveCodeBench, etc), at the same cost. Interactive site and academic paper released.
Key findings:
- Quoted API pricing is mostly fiction: on real workloads Kimi costs more than advertised, GPT-5.4 runs cheaper
- Hidden variance is the silent killer: running identical prompts 10 times shows Qwen3.7 Max 96.1% reliable, Claude Opus 4.6 94.4%, GPT-5.5 93.5%
- In agent workflows one random failure breaks the whole chain — don't bet on a single API
Related event: Martian's AI Frontier: Routing Across 44 LLMs Cuts Errors at Equal Cost(8 posts)→
More from coding & agent
- MCP Platform NitroStack Wins Product Hunt Product of the Day, Focuses on Retention — EstablishmentFun4373 · 2026-10-21
- CodeRabbit Ships 5,000 PRs a Week With a Tiny Team, Hires by Strategy-Game Hours — omojumiller · 2026-10-21
- LukeW maps the five UI problems every AI-native company faces with agents — LukeW · 2026-10-21
- Claude Code over ADB guts a bloatware-laden Android TV, freeing ~90% more RAM — edwinarbus · 2026-10-21
- Developer uses Claude to fix his adware-slowed Xiaomi TV, releases tool for any Android TV — edwinarbus · 2026-10-21
- Open-weights Ornith-1.5-122B debuts with 84.5% SWE-bench Verified, runs on a single DGX card — _akhaliq · 2026-10-21