Production AI budgets include retries, routing, caching and observability—not just token prices
arx-go · reddit · 2026-07-22
The post argues that production AI costs are much bigger than a model’s per-token price.
It breaks the budget into several parts:
- LLM/API usage
- Retries and failures
- Routing across multiple models
- Caching, or the lack of it
- Embeddings and vector databases
- Guardrails and moderation
- Monitoring and observability
- Infrastructure and orchestration
The main point is that AI spending should be treated as a system-level budget, not a single line item. The author links to a short visual article explaining the idea and asks how other startups budget for AI workloads.
More from Infra
- One month fully local on a DGX Spark: daily tasks fine, code still needs Claude to check — michaelthatsit · 2026-10-27
- Cloudflare Workers adds spanContext, setStatus and updateName tracing APIs — irvinebroque · 2026-10-27
- M5 Max benchmark: Kapitsa hits 36.6 tok/s with KFlash, 28GB at 32K vs mlx-lm's 52GB — DotaMate · 2026-10-27
- Unsloth squeezes 428B MiniMax M3.1 Flash into 143.9GB 2-bit GGUF with 91.5% accuracy — danielhanchen · 2026-10-27
- Stranded compute, neocloud debt worries, and the rise of 'micro-neoclouds' — mobileraj · 2026-10-27
- Chutes roadmap: cut loss-making inference, push hourly rentals and edge nodes — jon_durbin · 2026-10-27