Running Kimi K3 on 8x B300: $190 per million tokens, full cost breakdown
OtherRaisin3426 · reddit · 2026-08-23
The author hosted the 2.8T-parameter Kimi K3 on Modal with 8x B300 ($56.79/hr), using vLLM with tensor parallel 8 and native MXFP4:
- Cold boot 27 min (1.56 TB load, JIT, 51 CUDA graph captures)
- TTFT 0.92–1.02 s, steady 92 tok/s decode, 83 tok/s average over 4 prompts
- $190 per million output tokens; a clean run costs $36 in GPU time; keeping it warm costs $1,363/day
As a comparison, Unsloth's 1-bit UD-IQ1S Dynamic GGUF (594 GB) fits 8x A100-80GB via llama.cpp at $19.99/hr (2.8x cheaper) — but delivers only 9 tok/s with 7–60 s TTFT, working out to $620 per million tokens (3.3x more expensive per token). Quality at 1-bit was fine (correct arithmetic, coherent prose). Full write-up with every flag, the Modal deployment file, and raw benchmark JSON on the linked blog.
More from Infra
- engy.ai processes 50B+ tokens in a day as open-model inference demand surges — const_reborn · 2026-10-16
- New inference engine 'e' hits 10,500 tok/s on a single DGX Station — brandon_galang · 2026-10-16
- SGLang ships Day-0 support for FLUX 3 Image Klein: 10 reference images, BF16/FP8/NVFP4 variants — ying11231 · 2026-10-16
- NVIDIA's Nemotron 3.5 Super VL hits CoreWeave: 12B-active multimodal MoE at $0.25/M input tokens — wandb · 2026-10-16
- Chip equipment beats bull case: 2026 WFE seen at $153B, $278B by 2030 — BenBajarin · 2026-10-16
- Steam now launches in 5.23s, beating Epic's 5.47s — and vLLM gets roasted for slow startup — AlpinDale · 2026-10-16