Martian's model routing cuts errors 46% vs best single LLM at 85% lower cost

rohanpaul_ai · x · 2026-09-04

Martian released AI Frontier, a dashboard for comparing LLMs by task, quality, actual cost, and reliability, arguing that picking one "best model" is the wrong unit of optimization.

Key claims:

Across 16 widely used benchmarks (TerminalBench, LiveCodeBench, etc.), oracle routing cut average error 54% at matched cost;
It beat the single best LLM by 46% fewer errors, or matched top-model quality at 85% lower API cost;
Every benchmark misses the majority of model capabilities, so combining models can yield a better mix of quality, cost, and reliability.

The tool also accounts for output length, reasoning behavior, retries, and consistency. An interactive site and an academic paper are available.

Original post →

Martian's model routing cuts errors 46% vs best single LLM at 85% lower cost

More from coding & agent