Building a multi-model agent gateway: streaming and tool calls are the real hard part
Amendxd · reddit · 2026-09-21
The author's team built a small gateway for their multi-model agent workflows and learned that provider switching is an integration problem, not a model-selection problem.
Pain points cluster at the boundary:
- SSE chunks don't align with message/JSON boundaries
- OpenAI toolcalls vs Claude tooluse blocks carry the same info in different shapes
- Providers return inconsistent status codes, error bodies, and finish reasons
- Per-project keys and billing views make usage accounting hard to reconcile
The gateway exposes two compatible paths: OpenAI-compatible Chat Completions and Claude Messages-compatible endpoints, while normalizing SSE events, translating tool-call formats, unifying errors, and handling routing plus usage/balance tracking in one console.
The hardest part isn't the happy path: partial-stream reconnection, preserving raw provider events for debugging, and keeping retries/fallbacks from corrupting cost accounting still need the most testing.
More from coding & agent
- Free DiffusionGemma endpoint returns sentence probabilities in ~230ms, OpenAI-compatible — victormustar · 2026-09-21
- Mozilla AI runs a local 30B model end-to-end to open a real bugfix PR, fully offline — mozilla-ai · 2026-09-21
- GPT-6 plans and codes a game, pulls production 3D assets via Hyper3D Rodin MCP — CodeByPoonam · 2026-09-21
- Dev demo: agent codes a game and generates all 3D assets itself via Hyper3D Rodin MCP — CodeByPoonam · 2026-09-21
- mcp-server-dns: Zero-config MCP server for DNS, WHOIS and domain checks — modelcontextprotocol · 2026-09-21
- Each MCP tool passes solo tests, but the agent's full workflow breaks on state passing — Stock-Pumpkin-8859 · 2026-09-21