LLM Inference Costs Drop Below $3 with B200s, Yet API Prices Stay High
AccBalanced · x · 2026-07-30
A recent post highlights that serving large models is becoming significantly cheaper. Even with expensive B200 GPUs, vLLM, and without major optimizations, inference costs drop below $3. However, these savings aren't reflected in API endpoints, leaving users frustrated by the lack of affordable services like a $5 Kimi endpoint.
Related event: GPT-5.6 Luna Price Cut 80%: Benchmark Shows Superior Cost-Efficiency(15 posts)→
More from Infra
- vLLM serves Beam at 1,769 tok/s on 4x B300 NVFP4, publishes Docker image and recipes — vllm_project · 2026-10-16
- SkyRL ships Day-0 RL support for Beam with +62% LoRA decode throughput in vLLM — vllm_project · 2026-10-16
- DFlash2 delivers up to 3.03x decode-only speedup for Beam across five benchmarks — vllm_project · 2026-10-16
- vLLM's Beam serving kernels cut GB200 forward time from 8.15 to 6.47 ms at TP8 batch 128 — vllm_project · 2026-10-16
- Inside Beam: 8-of-256 expert routing across 52 layers, sliding window plus NoPE attention — vllm_project · 2026-10-16
- vLLM ships Day-0 support for Reflection-Beam-1.0, a 501B-param MoE with 1M context — vllm_project · 2026-10-16