Ex-HFT engineer open-sources a C++20 io_uring LLM gateway with 0.18ms median streaming overhead
lluisantoni · reddit · 2026-09-17
A routing engineer at Kottos AI open-sourced llmbridge (Apache 2.0, github.com/kottos-ai/llmbridge), a gateway that accepts OpenAI-compatible clients and translates requests/responses to Anthropic, Gemini or Cohere, including SSE streaming and Anthropic tool calls. Coming from electronic trading, the author built it like a market gateway targeting microsecond-level overhead:
Design: zero third-party runtime dependencies by default (only optional TLS pulls in OpenSSL); hand-written JSON parser/serializer limited to the handled formats, with the DOM holding stringviews into the input buffer and passthrough fields emitted verbatim; a zero-copy, zero-allocation HTTP/1.1 parser; iouring by default with an epoll fallback — 14 method pairs duplicated across backends, with a CI script enforcing ep/ur naming to block cross-backend calls.
Surprises caught by measurement:
- The memset wiping API keys from reused buffers was optimized away at -O2; now explicitbzero or a memset behind a volatile function pointer.
- Building 10 MB request bodies in fresh std::strings cost 2.6ms in page faults; buffers are now reused.
- Six header walks per request cost 40us at p99; merged into one pass.
- Quotes appear roughly every 8 bytes in streamed JSON chunks, so stringview::find wasted 65us per streamed request retrying comparisons; switched to memmem.
Performance: on Jakub A. Wąsek's Enterpilot benchmark (laptop, mock upstream, one worker, no translation), median added latency was 0.07ms non-streaming and 0.18ms streaming, excluding inference. Network parts are Linux-only; the translator is plain C++20 and builds on macOS. Still missing: vision, Gemini/Cohere streaming, and Anthropic-client-to-OpenAI-upstream translation.
More from coding & agent
- Microsoft's MemSteer turns unlabeled user histories into agent memory supervision — MicrosoftResearch · 2026-11-02
- She fired herself from her own Instagram: AI agent runs the account, views up 82.3% — heyshrutimishra · 2026-11-02
- Sourcegraph: AI writes code faster than teams can review it, driving more outages — AI Engineer · 2026-11-02
- Tsinghua & Microsoft's HarnessMM cuts agent token use by up to 87.93% while raising success rate — rohanpaul_ai · 2026-11-02
- GitHub launches Agent HQ: Anthropic, OpenAI, Google coding agents in Copilot — emmanuelvivier · 2026-11-02
- DeepSeek open-sources Super-Agent, a coding agent that reads team chats to minimize human input — emmanuelvivier · 2026-11-02