A multi-provider LLM gateway and the billing bugs that mattered
MTreeAI · reddit · 2026-07-20
The author describes the architecture of a multi-provider LLM gateway and the bugs they found while making billing trustworthy.
Architecture
- About 18 LLM providers are routed behind one API.
- Most providers collapse into an OpenAI-compatible dispatch path; a smaller set (Anthropic, Gemini, Cohere, Replicate) needs custom handling.
- Tool-calling adapters are separated into a pure-function layer that normalizes differences in tool schemas, toolchoice, parallel calls, usage parsing, and seed handling.
Billing design
- All requests are normalized to USD per token at record time.
- Every call goes through a single billing chokepoint:
- charge a small preflight amount under a row lock,
- make the API call,
- reconcile estimated vs. actual usage and refund the difference.
- Local models bill at zero.
Three money bugs
- A refund path could mint credits because failed requests sometimes refunded more than was actually charged.
- Streaming refunds were skipped on client disconnect because asyncio raises GeneratorExit, which was not caught by except Exception.
- Two functions each wrote a ledger row, causing double charges; the fix was to keep one source of truth.
Takeaway
Correct, centralized metering beat clever cost-routing. The “cheapest-provider” router is turned off by default.
More from coding & agent
- GPT-6 Astra beats Factorio with enemies in 44 in-game hours at ~$4,500 API cost — liminal_bardo · 2026-09-11
- 105 hidden bugs, 2 repos: DeepSeek V4.1 Flash fixes 24 at $1.80 vs Opus 5's 27 at $51.33 — ChartsJournalX · 2026-09-11
- Investment Analyst Asks How to Build a Claude-Based Diligence Agent Stack — Careless_Tie2286 · 2026-09-11
- Treating agents like 50 First Dates: a 3-layer context system so every conversation doesn't start from zero — evielync · 2026-09-11
- Running the Firefox MCP on Android via Termux, ngrok, and mcp-proxy — Nervous-Strain7544 · 2026-09-11
- SmolVM open-sources persistent computer infrastructure for agents that outlive chat sessions — aniketmaurya · 2026-09-11