PrefixRelay: open-source proxy gives Cursor 97%+ KV cache hits on local models
jjusko20 · reddit · 2026-11-12
Developer jackjusko open-sourced PrefixRelay, an OpenAI-compatible proxy layer for Cursor addressing four pain points of running local/cheap models: it mutates responses to keep prefix caches valid (97%+ hit rate across thousands of requests, vision-safe); presents replica pools of multiple GPU instances as a single model so Cursor can spawn subagents with intelligent prefix-based routing; runs its own context-aware compaction since Cursor hardcodes 200k context for unknown models; and bypasses Cursor's localhost restriction via a bundled Cloudflare tunnel quick start. Repo: github.com/jackjusko/PrefixRelay
More from coding & agent
- Harvard's new CS2680 walks students from Claude Code down the LLM serving stack — udmrzn · 2026-11-12
- Local vs Cloud AI Coding Agents: Benchmarks, Hardware Costs, Performance — luisdans · 2026-11-12
- The simplest heuristic for agent UX: expect it to behave like a capable coworker — andreisavu · 2026-11-12
- Agent Substrate 1.0 GA: microVM-sandboxed agent runtime with zero ambient credentials — rakyll · 2026-11-12
- Dev builds sketch-guessing app with new Gemini Decisions API, guesses terrible drawings — fofrAI · 2026-11-12
- Together Compiler: agent-driven continual compilation for AI inference — togethercompute · 2026-11-12