SemiAnalysis: vLLM hits 1.3B tokens per $ TCO, 4x better than B300 aggregated serving
woosuk_k · x · 2026-10-18
SemiAnalysis reports that vLLM and inferact achieved 1.3 billion tokens per dollar of TCO (hyperscaler volume pricing) via disaggregated serving on AgentX — 4.01x better than the B300 aggregated configuration. Testing DeepSeek V4.1 Flash, the optimal 10×DEP2+1×DEP16 setup reached 1.3B tok/$ at 100 tok/s/user P90 interactivity, while B300 TP4 aggregated configs offer higher per-user speed but an order of magnitude fewer tokens per dollar. Key takeaway: P/D disaggregation massively beats aggregated serving on cost efficiency.
More from Infra
- TSMC: 'No intention to leave any food on the table for others' — Beth_Kindig · 2026-10-18
- Running Fish Audio Drama 3 on a 16GB AMD GPU: Red Curtain ports the 35B MoE that officially wants 48GB — fechyyy · 2026-10-18
- DeepInfra cuts DeepSeek V4.1 Flash price 51% as agent demand surges — gharik · 2026-10-18
- a16z avoids neoclouds over debt risk; Bittensor pitched as the answer to CoreWeave's model — markjeffrey · 2026-10-18
- Local AI users rant: 16GB VRAM GPUs can't run any decent modern models anymore — 2MuchNonsenseHere · 2026-10-18
- Meta lays out 448G in-rack interconnect roadmap: PAM6 candidate, active copper cables stretch reach — StewartalsopIII · 2026-10-18