Koi Paper: Self-Calibrating Planner Boosts Heterogeneous LLM Fleet Goodput 1.18x, Cuts $/token 20%
burny_tech · x · 2026-10-23
Researchers from Tandemn, UIUC, UC Berkeley, NYU, ByteDance and NVIDIA present Koi, a self-calibrating cluster planner for heterogeneous multi-cloud LLM inference fleets, where deployment requires jointly choosing accelerators, parallelism, engine config and replicas among 140 variables. Koi uses a Recursive Language Model hierarchy to propose hypotheses while deterministic tools predict serving behavior, score candidates and control placement, all grounded in a causal graph linking decisions, runtime mediators (KV-cache pressure, communication overhead) and outcomes. Each deployment updates scoped causal relationships, building an inference playbook. On real heterogeneous clusters Koi achieves 1.18x greater goodput than the strongest baseline and 20% lower $/token; ablations show composite scores of 35.3 (pure LLM) vs 51.6 (+deterministic tools) vs 70.8 (full planner).
More from Infra
- MiniMax M3.1 Flash launches day-zero on Modular Cloud, tops open models in cowork automation — MiniMax_AI · 2026-10-23
- NVIDIA's $1.2M HGX Rubin NVL8 hits GTC Berlin, promising 10x tokens per megawatt over B200 — thursdai_pod · 2026-10-23
- Amazon scraps plans for four data centers on $427M former GWU Ashburn campus — GaryMarcus · 2026-10-23
- Developer platforms raided for AI inference keys: Railway, Fireworks, LiteLLM all hit this month — ccerrato147 · 2026-10-23
- Inco Open-Sources DFlash 2 Drafter: Lossless 3.5× Speedup for MiniMax M3.1 Flash — Lianhuiq · 2026-10-23
- Cloudflare CEO: agent traffic will be 1000x human traffic in 5 years, internet needs micropayments — 4KTV · 2026-10-23