A multi-provider LLM gateway and the billing bugs that mattered
MTreeAI · reddit · 2026-07-20
The author describes the architecture of a multi-provider LLM gateway and the bugs they found while making billing trustworthy.
Architecture
- About 18 LLM providers are routed behind one API.
- Most providers collapse into an OpenAI-compatible dispatch path; a smaller set (Anthropic, Gemini, Cohere, Replicate) needs custom handling.
- Tool-calling adapters are separated into a pure-function layer that normalizes differences in tool schemas, toolchoice, parallel calls, usage parsing, and seed handling.
Billing design
- All requests are normalized to USD per token at record time.
- Every call goes through a single billing chokepoint:
- charge a small preflight amount under a row lock,
- make the API call,
- reconcile estimated vs. actual usage and refund the difference.
- Local models bill at zero.
Three money bugs
- A refund path could mint credits because failed requests sometimes refunded more than was actually charged.
- Streaming refunds were skipped on client disconnect because asyncio raises GeneratorExit, which was not caught by except Exception.
- Two functions each wrote a ledger row, causing double charges; the fix was to keep one source of truth.
Takeaway
Correct, centralized metering beat clever cost-routing. The “cheapest-provider” router is turned off by default.
More from coding & agent
- Dense instance segmentation via Muse Wonder + SAM 3.1 on Meta's Model API — nikhilaravi · 2026-10-26
- Security tip: use AI agents to build a NIST CSF knowledge base for incident response — prasanna_says · 2026-10-26
- Google Cloud Adds Self-Learning Context Enrichment to Knowledge Catalog for AI Agents — rseroter · 2026-10-26
- Cloudflare Previews Spend Limits for Workers and Durable Objects, Early Access Open — ritakozlov · 2026-10-26
- Shopify design lead on AI-era workflow: designers are becoming full-stack editors — KlausCodes · 2026-10-26
- Nace AI launches Drex for Search, cutting agent steps 62% and tokens 56% — testingcatalog · 2026-10-26