Three myths of hosted LLM inference: sticker prices, interchangeable endpoints, and self-hosting
TangeloOk9486 · reddit · 2026-09-27
A practitioner distills the recurring myths in hosted-inference debates:
- Price per token is the comparison. In reality cache behavior, batch shape, and minimums decide the real bill; sticker rate is just the opener.
- Every OpenAI-compatible endpoint is interchangeable. Model freshness, cold starts, rate limits, and support quality are the actual product — API shape is table stakes, whether it's maker-direct or a flat host like deepinfra.
- Self-hosting always wins at scale. It wins only past sustained utilization; below that, ops time plus idle VRAM quietly costs more than metered inference.
Providers thrive because most teams would rather pay per token than hire for GPUs.
More from Infra
- Random Attention: Salesforce and UIUC find random KV cache eviction rivals handcrafted signals — jiqizhixin · 2026-09-27
- Laptop engine streams a 35B model from SSD at 9.4 tok/s, beating GPT-OSS 20B — ImBadGuyInEveryStory · 2026-09-27
- World's fastest panel QR factorization on B200: how a GPU MODE contestant cracked chained dependencies — A_K_Nain · 2026-09-27
- TensorSharp open-source engine adds mixed document/image/video/audio inputs per request — fuzhongkai · 2026-09-27
- Pro-data center rally clashes with protesters as scholar defends AI infrastructure — neil_chilson · 2026-09-27
- Kafka in production: partition skew and rebalancing pauses bite hard — goyalshaliniuk · 2026-09-27