Dev builds Rust desktop API gateway unifying OpenAI/Anthropic APIs on one GPU
rootshelldev · reddit · 2026-10-07
A single-GPU home developer built a Rust + Tauri + Leptos desktop API gateway to tame model/API fragmentation while staying compatible with clients that only support official Anthropic/OpenAI upstream APIs.
Highlights:
- Text, embeddings, audio and image APIs routed to cloud or llama.cpp / audio.cpp / stable-diffusion.cpp containers, built from git including MRs/PRs;
- One container per model with VRAM-aware scheduling, fallbacks, and alias-based model chains (e.g. auto-reload with larger context on demand, cloud routing past a context threshold, high-quant→low-quant tiers);
- Tray-triggered "GPU hold" mode that unloads models for gaming;
- OpenAI-compatible /v1/realtime websocket pipeline (VAD → ASR → LLM → TTS), each step selectable local or cloud, with barge-in and session management;
- MCP gateway running servers in podman containers with scoped tokens, plus an admin MCP so coding agents can configure everything; a context7-like docs MCP for missing/stale libraries with ingestion into vector storage;
- Detailed metrics, budgets, cost comparisons, and an integrated test chat UI.
More from coding & agent
- "Fix your failing self until the task completes": a developer's go-to Codex prompt — jdjohnson · 2026-10-07
- SEO Workflow: Run Grok 4.7 Overnight on 10K+ Pages to Flag Stale Content — gaganghotra_ · 2026-10-07
- Hark Agent Early Access Review: Best-in-class UI, Own Cloud Computer for Lightning-fast Tasks — adcock_brett · 2026-10-07
- threepointone recommends krazam's 'You Are An Expert Software Engineer', likening it to qntm's 'The Difference' — threepointone · 2026-10-07
- Wattenberger: codebases must distinguish specified vs hinted vs agent-assumed — Wattenberger · 2026-10-07
- OpenAI's Decisions API now takes image input — one creator picks YT thumbnails for $0.13 — stevenheidel · 2026-10-07