Fireworks: routing 18 models per task hits 97.6% solve rate at $1.88 vs best single model's 74.1% at $6.52
sophiamyang · x · 2026-09-22
Fireworks AI ran 18 models across 113 real coding tasks on DeepSWE v1.1 and found that routing each task to its best-fit model yields 97.6% solve rate at $1.88 per task, versus 74.1% at $6.52 for the best single model (GPT-6 Astra) — 23 points better at under a third of the cost.
Key points:
- The figure comes from an oracle router (best model chosen in hindsight), measuring latent capability in the model pool rather than real router performance.
- The hard part is predicting the right model before work starts; a wrong choice costs more than the savings.
- Policy: one model fixed per task, no mid-session switching; 4 rollouts per model-task pair.
- Fireworks is shipping FireRouter, arguing "the frontier isn't a model, it's a router."
More from coding & agent
- 引用讨论:给 agent 团队堆算力可能是新 scaling law — burny_tech · 2026-09-22
- Teknium merges fix for Hermes Agent Desktop failing to resolve model-provider plugins — Teknium · 2026-09-22
- 90s multi-agent systems offer a lesson: model memory as belief, not centralized truth — nptacek · 2026-09-22
- Grok 4.7 jumps to 46.3% on CursorBench and 64% on EEBench, keeping the same $2/$6 per million token pricing — FinanceYF5 · 2026-09-22
- OpenCode 2.0 Blog Reveals an Agent That Rewrites Itself Live During Sessions — aidenybai · 2026-09-22
- Tests Passing ≠ Ready to Ship: Reddit Debates Whether AI Coding Agents Should Merge Their Own PRs — Fantastic-Sleep-3352 · 2026-09-22