6 bugs that break naive LLM budget caps under concurrency—and their fixes
MaverikSh · reddit · 2026-10-24
Capping LLM spend by checking spent-vs-limit breaks in production with parallel workers, streaming and retries. The author walks through 6 repeatable failure modes and fixes:
- Check-then-act race: 50 workers all pass the check; fix by making check+reserve atomic (Redis Lua or conditional Postgres UPDATE), reserving worst-case cost upfront.
- Floats for money: drift over millions of adds; use integer micro-dollars and HINCRBY.
- Leaked reservations: crashed workers lock budget forever; give reservations IDs + expiry (sorted set), run a sweeper, settle by reservation ID.
- Cut-off streams: token usage never arrives on disconnect; settle at reserved estimate or count streamed output yourself.
- Nested budgets deadlocking: reserve all levels (org/team/key) in one Lua script with hash tags, or lock rows in fixed order in SQL.
- Period rollover: store the period key on the reservation and settle against it.
Plus a worst-case estimator (local token count × prices × 1.1 margin, since tiktoken miscounts for Claude/Gemini) and a middle-ground fallback when the budget store is down: allow calls with a small per-key outage-only allowance.
More from coding & agent
- Google's playbook: evaluating AI agents beyond the vibe check — bibryam · 2026-10-24
- Running CI weekly on dotfiles setup.sh inside a fresh Ubuntu container to catch breakage — 4310sy · 2026-10-24
- Codex CLI's hidden remote-control feature praised as better than Claude Code's — antirez · 2026-10-24
- PWAs can't use foldable screens on the Duo yet, but two Chrome flags unlock the APIs — mitsuhiko · 2026-10-24
- Reddit user asks which no-code AI agent builders actually work beyond the demo — Austin_Ox23 · 2026-10-24
- Dev compares Claude vs OpenAI's Astra: Claude fixes its own mistakes, Astra spins for days — StewartalsopIII · 2026-10-24