Baseten Breaks Down Serving 2.8T-param Kimi K3 at Scale on Blackwell
thursdai_pod · x · 2026-08-09
Philip Kiely from Baseten provided an in-depth breakdown of how to serve massive 2.8-trillion parameter models like Kimi K3 at scale using Blackwell GB300s.
To achieve this highly challenging deployment, the team performed deep optimizations on the underlying inference stack and contributed their improvements back to the open-source community, enhancing the capabilities of vLLM and SGLang for ultra-large models.
More from coding & agent
- Constraints you force on models today will become tech debt in months, warns dev — zack_overflow · 2026-10-16
- Liquid AI tutorial: fine-tune your d1-3B decision model in 30 minutes on a free GPU — helloiamleonie · 2026-10-16
- Alibaba open-sources OpenGCUA, a full-stack generalist computer-use agent; +19.6 points on OSWorld-v2 across 8 models — Tongyi-MAI · 2026-10-16
- BotBeat: a social network where all users are AI agents, exposed via 32 MCP tools — Dismal_Piccolo4973 · 2026-10-16
- DKGI paper: agents write their own tools to search knowledge graphs, beating baselines by 6.7 Hit@1 — yuz9yuz · 2026-10-16
- Dev open-sources lgtmaybe: running PR code reviews on a local 27B model via hunk splitting — mattjcoles · 2026-10-16