Agent-run OpenRouter routing eval: cheapest providers are unreliable, fastest nearly as cheap with caching
MikePFrank · x · 2026-11-23
Mike Frank had his Qwen3.8 27B agent "Lantern" work with Astra 6.2 to evaluate cost, reliability and speed across OpenRouter routing choices over 7 days.
Key findings:
- Cheapest providers tend to be unreliable: Darkbloom ($0.046/M) mangled tool calls and was excluded; Near AI ($0.062/M) had an empty-reply turn.
- With prefill caching accounted for, the fastest route is nearly as cheap as the cheapest: Nebius fastest route at $0.396/M with 100% reliability vs $0.254/M for cheapest.
- Cost per successful turn: cheapest 1.99¢ (8 success/5 failed), fastest 2.38¢ (15/0), default 5.39¢.
- Lantern also spotted and fixed a real billing bug where providerstats() showed a minimum per route; it verified exact averages through turn 170 and confirmed OpenRouter charged $0.0143 for a timeout.
More from Infra
- Anthropic's on-site power is 80% simple-cycle gas; health damages estimated at $2-3B per year — petrusenko_max · 2026-11-23
- Inkling-2-Mini GPTQ Quant Ships 524K-Token Context in One 18.3GB File on a Single 24GB Card — W61k3r · 2026-11-23
- Dev: 99% of people shouldn't self-host local LLMs, and he only does it for the hardware — gnukeith · 2026-11-23
- Samsung Labs open-sources LittleBit, compressing LLM weights to 0.1 bits — MikePFrank · 2026-11-23
- Tech Sells Data Centers as Local Wins, But Jobs Are Just Guards and Remote Engineers — feregri_no · 2026-11-23
- RSIBench-Infra from ByteDance Seed and Tsinghua tests if agents can optimize production-scale AI infrastructure — geoffwolfe · 2026-11-23