How do teams actually control LLM inference costs in production? A Reddit thread asks
Ok_Philosophy_4031 · reddit · 2026-09-27
A developer asked how teams with real production LLM spend actually control inference costs once past the MVP stage.
Beyond the obvious playbook — switching to cheaper models, prompt/token reduction, caching, batching, routing, fine-tuning small models, self-hosting — the poster wants practice, not theory:
- Which optimizations actually moved the bill, and which weren't worth the engineering complexity?
- How much do reliability/quality constraints block switching to cheaper models?
- Should you optimize $/token, $/request, or $/successful task?
- How much spend goes to hard reasoning vs. repetitive extraction, classification, routing and scoring?
- Do teams actively replace LLM calls with conventional software/ML where possible?
The author suspects a big gap between inference optimization in blog posts and what teams are actually willing to maintain in production.
More from coding & agent
- Why AI agents may not go mainstream this decade: most lives aren't complex enough — menhguin · 2026-09-27
- A practical AGI test: replace one human validating 150,000 document conversions — burkov · 2026-09-27
- Talk on self-improving agents with reflective optimization: GEPA, gskill, fast-slow training — CShorten30 · 2026-09-27
- Microsoft president Jeff Teper shares how he uses Autopilot for work routines — heyneighbor · 2026-09-27
- Same prompt weekly for a quarter: how a prod agent's policy compliance drifted until it broke — IsomuraArganee_95 · 2026-09-27
- Diplomacy comes to Multi-Agent Arena: test your social strategy against frontier LLM agents — ycombinator · 2026-09-27