Batch APIs: the 50% LLM discount only pays off after model choice and prompt caching
arpit_bhayani · x · 2026-11-13
Arpit Bhayani breaks down how to think about LLM batch APIs beyond the headline 50% discount: real savings = 50% × the share of traffic that can tolerate up to 24-hour latency (60% eligible → 30% saved).
Key points:
- Discounts stack: batch pricing applies after prompt caching, so better caching shifts savings toward output tokens — output-heavy workloads benefit more than the rate card suggests.
- Decision order: pick the right capability tier first (real-time Haiku can beat batched Sonnet on cost and latency), enable caching next, only then ask what can wait 24 hours.
- Adoption checklist: is eligibility high enough to justify queues/polling/webhooks/failure handling; can you live with the tail latency (need a real-time fallback for deadlines); would a cheaper tier suffice.
Conclusion: batch is almost always a good lever, but it comes last.
More from Infra
- Report: The End of the Jensen Huang Era at Nvidia After 30 Years — examachine · 2026-11-13
- Nvidia pressured to boost 25% residual value backstop on planned $500B AI financing vehicle — dinabass · 2026-11-13
- A GPU deep-dive for AI engineers: token flow, memory math, quantization and multi-GPU sharding — kmeanskaran · 2026-11-13
- CoreWeave Q3 revenue hits $3.66B, up 168%; signs multi-year Wayve training deal — thursdai_pod · 2026-11-13
- 67B MiniMax H3-Image runs on a 16GB MacBook Air via MLX with ~85% less peak memory — FlimsyBrief4232 · 2026-11-13
- LeapQuant quantizes linear attention states to 8/4-bit training-free with 2x throughput — zealcaiden · 2026-11-13