PyroDash cuts inference cost 96% by having a 4B model call the big one only when needed
jiqizhixin · x · 2026-09-22
Pyromind releases PyroDash, a token-level small-large model collaborative inference framework.
- Key idea: instead of routing whole queries to a big model, a 4B small model decides at token granularity when to call for help — on average once every 100 problems.
- Training: RL with reward = accuracy − λ × normalized cost (absolute cost over the all-big-model baseline), using the EasyHard-24k dataset as the training ground.
- Results: output price as low as $0.08 per million tokens (vs $75 for top-tier models), 96% lower inference cost, and it beats using the big model alone across five math reasoning benchmarks.
- Authors' insight: RL can train not just a model but an entire system to divide labor between quality and cost.
More from Infra
- COLIBRI: pure-C zero-dep engine streams 2.8T-param MoE models from disk on consumer hardware — bibryam · 2026-09-22
- Sentdex benchmarks openjev: 169ms on Dell GB10 vs 137ms on RTX 3090 — Sentdex · 2026-09-22
- Underdog's Husky Inference Engine Claims 4.5x Speedup Over MLX, 730 tok/s on MacBook — jimmykoppel · 2026-09-22
- Full-Parameter RL on TPUs: peano_ai Runs 310B MiMo-V2.6 Across 1,000+ TPUs — simonguozirui · 2026-09-22
- Cloudflare Python Workers go generally available after two-year preview — Simon Willison · 2026-09-22
- Fighting AI crawler traffic: beyond Turnstile, Cloudflare's AI Labyrinth as an option — fforres · 2026-09-22