Open-source Zilla adds LLM gateway support: preserving API fidelity across providers
jkriket · reddit · 2026-10-28
The open-source project Zilla (aklivity/zilla) added LLM gateway support. The author argues most gateways normalize everything into an OpenAI-compatible format, silently dropping native features like Claude's prompt caching, extended thinking, and streaming tool calls — while passthrough workarounds bypass the governance policies the gateway exists to enforce.
Zilla's approach:
- A common representation for shared concepts (messages, tools, usage) while retaining provider-specific details
- Same-dialect traffic (e.g. Anthropic→Anthropic) keeps full fidelity: caching markers, thinking signatures
- Cross-dialect translation where possible, e.g. reasoning effort mapped between OpenAI and Claude
- Governance applies to every route; streaming works across dialects, including tool-call fragments and stream↔non-stream conversion
- Built on the same streaming engine long used for Kafka, HTTP, MQTT, and MCP
The open-source edition supports OpenAI, Anthropic, Gemini, AWS Bedrock, Google Vertex, and Azure Foundry.
More from coding & agent
- GitHub's Spec Kit pushes spec-driven development: 5 commands to plan before coding agents build — Arindam_1729 · 2026-10-28
- How LangChain built its design agent: taste comes from context, not the model — LangChain · 2026-10-28
- Dev predicts in 5 years we'll laugh at today's juggling of terminal and agent harness tabs — DavidWells · 2026-10-28
- Team reverse-engineers Claude Code's memory: simple design, with caveats — letandrewcook · 2026-10-28
- Multimodal GraphRAG models missing and contradictory evidence for storm damage assessment — JeremyCMorgan · 2026-10-28
- Vibe coding in real life: 20 hours of personal-project work done in 2 — gleech · 2026-10-28