AI Features in Prod: How Real Users Turn Token Costs into a Flamethrower
Busy-Yogurtcloset617 · reddit · 2026-08-12
A developer shared a lesson on "cost runaway" encountered after pushing an AI feature to production, sparking a discussion on controlling token spend.
- Cost Explosion: The AI feature, which looked fine in staging, caused costs to skyrocket with real users. Users frequently submitted overly long questions, pasted massive amounts of CRM text, and maintained multi-turn conversations without resetting context.
- Retrieval & Context Flaws: The retrieval layer duplicated identical snippets, and combined with giant system prompts and full conversation history, the token count per request became massive.
- Optimization Paths: The author notes that fixing this requires more than just reducing tokens. It demands better text chunking, deduplicating retrieved text, capping history length, and balancing the trade-off between latency and accuracy.
More from coding & agent
- Developer Shares Workflow for Building an AI Chief of Staff Using MCP — ayushtweetshere · 2026-08-12
- Developer Finds OpenAI Codex Uses `claude -p` to Spawn Sub-Agents — cramforce · 2026-08-12
- Developer Asks AI to Fix Bug, AI Rates Its Own Confidence at 50% — _jaydeepkarale · 2026-08-12
- Exploring Business Models Around Claude Code and Codex Plugins — doooyle · 2026-08-12
- Building a Minimalist Local Agent Stack: SearxNG Search and Reasoning Budget Control — use_your_imagination · 2026-08-12
- Sequoia Shares Harvey's Playbook: Building Research-Level Legal Agents on a Budget — Scobleizer · 2026-08-12