LLM routers audited: none of 7 beats a static mid-priced model at 85.3% for $0.0043
dl_weekly · x · 2026-11-04
A pre-registered, cost-aware audit of 7 commercial and open-source LLM routers (AutoBench Router Benchmark, 252 prompts, 20-model pool) finds none beats always calling one mid-priced model — GPT-5.6 Luna scores 85.3% at $0.0043/question. The theoretical prize is real (100% Delta-Hit at $0.0018 for perfect routing vs 87.5% at $0.079 for always-top-model), but capture is zero: across 17 settings, 5 collapsed to a single model taking ≥90% of traffic, and no setting ever beat the static policy in 10,000 bootstrap resamples.
More from Models
- Weight-editing author: gains aren't real knowledge — Gemma math didn't improve, but Qwen3 issue is real — lateinteraction · 2026-11-04
- rasbt questions Qwen3 weight-editing gains: 50 GRPO steps already lift MATH-500 from ~15% to ~50% — rasbt · 2026-11-04
- NVIDIA releases Dory1.0-4B, a looped recurrent LM beating larger models — nvidia · 2026-11-04
- New GDPx Benchmark: Frontier Models Score Near 0% on Full Legal Jobs — jkubicki · 2026-11-04
- Claude Shows Self-Awareness About Its Copyright Lyric Refusals — Angaisb_ · 2026-11-04
- Claude Opus 5 surgically edits ~1% of ProteinMPNN weights into SolubleMPNN-like behavior — AllThingsApx · 2026-11-04