Stop sizing LLMs by static leaderboards; compute the API-vs-self-host breakeven instead
zackku · reddit · 2026-10-27
The author argues leaderboard scores (Chatbot Arena Elo, GPQA) stop mattering in production, where two infrastructure bottlenecks dominate:
- API vs self-hosting breakeven: at what monthly token volume and prompt-cache hit rate does renting Spot GPUs with quantized weights beat paying OpenAI or Anthropic.
- Real VRAM sizing: weights are only half the story; once context windows and KV cache overhead are included, standard sizing charts break down and vLLM/Ollama hit unexpected OOMs.
He built a zero-login FinOps workbench covering 280+ models that lets you: drag sliders for monthly token volume and cache hit rate (0%-95%) to simulate token arbitrage; compute VRAM across FP16/FP8/INT8/Q4 with one-click copyable vLLM and Ollama launch commands; and model multi-agent routers that triage with smaller models before escalating to frontier ones, typically saving 60%-80%.
He closes by asking production users how they calculate token spend at scale and whether they default to APIs or self-hosting.
More from Infra
- Lovable invokes 753.6B Cloudflare Workers monthly, migrated from Next.js to TanStack Start — ritakozlov · 2026-10-27
- Prague lab BottleCapAI releases ThinkingCap models cutting thinking tokens up to 69% with negligible accuracy loss — JFPuget · 2026-10-27
- Local CPU vector mesh beats BM25 routing 100k tools: 70.7% top-1 at 9.6ms — Key_Coconut_9269 · 2026-10-27
- 4 coding agents on one RTX 3090: custom CUDA engine hits ~290 tok/s with Qwen3.8-27B — Adorable_Weakness_39 · 2026-10-27
- Google ships llm-d 1.0, an open Kubernetes-native inference stack, demoed on 1,000 agents — dl_weekly · 2026-10-27
- Transformers v5.21 ships FP8 KV-cache, MoE sharding, and new TTS models — LysandreJik · 2026-10-27