A multi-provider LLM gateway and the billing bugs that mattered
MTreeAI · reddit · 2026-07-20
The author describes the architecture of a multi-provider LLM gateway and the bugs they found while making billing trustworthy.
Architecture
- About 18 LLM providers are routed behind one API.
- Most providers collapse into an OpenAI-compatible dispatch path; a smaller set (Anthropic, Gemini, Cohere, Replicate) needs custom handling.
- Tool-calling adapters are separated into a pure-function layer that normalizes differences in tool schemas, toolchoice, parallel calls, usage parsing, and seed handling.
Billing design
- All requests are normalized to USD per token at record time.
- Every call goes through a single billing chokepoint:
- charge a small preflight amount under a row lock,
- make the API call,
- reconcile estimated vs. actual usage and refund the difference.
- Local models bill at zero.
Three money bugs
- A refund path could mint credits because failed requests sometimes refunded more than was actually charged.
- Streaming refunds were skipped on client disconnect because asyncio raises GeneratorExit, which was not caught by except Exception.
- Two functions each wrote a ledger row, causing double charges; the fix was to keep one source of truth.
Takeaway
Correct, centralized metering beat clever cost-routing. The “cheapest-provider” router is turned off by default.
More from coding & agent
- Hugging Face wraps 6-month Training Agents series with open recordings and code — SergioPaniego · 2026-10-26
- Agent verified 'success' after double-voting: a lesson in harness vs model bugs — TejasKumar_ · 2026-10-26
- diginomica: agentic automation hype is overdone, and 'The Hackening' of enterprise security has begun — jonerp · 2026-10-26
- 'Vibe verification': ~60% of AI-written formal specs prove the wrong property — basedjensen · 2026-10-26
- Devin launches Security suite powered by Claude Mythos with $100M+ credits — DevinAI · 2026-10-26
- HackBench injects reward hacks into agent trajectories to stress-test monitors — DanielKhashabi · 2026-10-26