Ex-HFT engineer open-sources a C++20 io_uring LLM gateway with 0.18ms median streaming overhead
lluisantoni · reddit · 2026-09-17
A routing engineer at Kottos AI open-sourced llmbridge (Apache 2.0, github.com/kottos-ai/llmbridge), a gateway that accepts OpenAI-compatible clients and translates requests/responses to Anthropic, Gemini or Cohere, including SSE streaming and Anthropic tool calls. Coming from electronic trading, the author built it like a market gateway targeting microsecond-level overhead:
Design: zero third-party runtime dependencies by default (only optional TLS pulls in OpenSSL); hand-written JSON parser/serializer limited to the handled formats, with the DOM holding stringviews into the input buffer and passthrough fields emitted verbatim; a zero-copy, zero-allocation HTTP/1.1 parser; iouring by default with an epoll fallback — 14 method pairs duplicated across backends, with a CI script enforcing ep/ur naming to block cross-backend calls.
Surprises caught by measurement:
- The memset wiping API keys from reused buffers was optimized away at -O2; now explicitbzero or a memset behind a volatile function pointer.
- Building 10 MB request bodies in fresh std::strings cost 2.6ms in page faults; buffers are now reused.
- Six header walks per request cost 40us at p99; merged into one pass.
- Quotes appear roughly every 8 bytes in streamed JSON chunks, so stringview::find wasted 65us per streamed request retrying comparisons; switched to memmem.
Performance: on Jakub A. Wąsek's Enterpilot benchmark (laptop, mock upstream, one worker, no translation), median added latency was 0.07ms non-streaming and 0.18ms streaming, excluding inference. Network parts are Linux-only; the translator is plain C++20 and builds on macOS. Still missing: vision, Gemini/Cohere streaming, and Anthropic-client-to-OpenAI-upstream translation.
More from coding & agent
- Diorama gives OpenAI Codex coding agents a visual office you can watch work in real time — davidfromkansas · 2026-09-17
- Code-first, UI on top: building bespoke brand design tools with AI — floguo · 2026-09-17
- Study of 7 models across Claude Code, Codex, Pi: harness barely affects success but swings cost — DavideCrapis · 2026-09-17
- AI trading bot built with Jev is down 85%, owner shrugs it off — generativist · 2026-09-17
- Redditor's 3-Day SoL-Pi Test: Memory Objects Save ~12k Tokens Per Tool Run — Garblyx · 2026-09-17
- Reviewing AI code through Steve Jobs' lens: unseen internals deserve beauty too — sergeykarayev · 2026-09-17