A multi-provider LLM gateway and the billing bugs that mattered
MTreeAI · reddit · 2026-07-20
The author describes the architecture of a multi-provider LLM gateway and the bugs they found while making billing trustworthy.
Architecture
- About 18 LLM providers are routed behind one API.
- Most providers collapse into an OpenAI-compatible dispatch path; a smaller set (Anthropic, Gemini, Cohere, Replicate) needs custom handling.
- Tool-calling adapters are separated into a pure-function layer that normalizes differences in tool schemas, toolchoice, parallel calls, usage parsing, and seed handling.
Billing design
- All requests are normalized to USD per token at record time.
- Every call goes through a single billing chokepoint:
- charge a small preflight amount under a row lock,
- make the API call,
- reconcile estimated vs. actual usage and refund the difference.
- Local models bill at zero.
Three money bugs
- A refund path could mint credits because failed requests sometimes refunded more than was actually charged.
- Streaming refunds were skipped on client disconnect because asyncio raises GeneratorExit, which was not caught by except Exception.
- Two functions each wrote a ledger row, causing double charges; the fix was to keep one source of truth.
Takeaway
Correct, centralized metering beat clever cost-routing. The “cheapest-provider” router is turned off by default.
More from coding & agent
- Tenable and AWS launch a Black Hat build event for open-source security agents and MCP servers — Dave_Maynor · 2026-07-22
- Codex helps build Valdiluce, an open-world game with climbing, gliding and gondolas — Dimillian · 2026-07-22
- HeyGen adds a media-sourcing skill for coding agents with 75k images and 10k tracks — HeyGen · 2026-07-22
- Agent search bottlenecks are now about variance, not raw latency — rohanpaul_ai · 2026-07-22
- LangSmith adds tracing for Pipecat, LiveKit, OpenAI Realtime, and Gemini Live — LangChain · 2026-07-22
- An MCP server signs every AI agent tool call into a verifiable Merkle chain — Funky_Chicken_22 · 2026-07-22