shad0w caches local LLM classification answers and serves repeats in microseconds
gauravc2796 · reddit · 2026-10-12
A developer released shad0w, a cache layer that sits in front of any OpenAI-compatible server (Ollama works out of the box: llm="ollama/llama3.1" or shad0w proxy --upstream http://localhost:11434/v1).
- It logs the model's classification answers, and after 1,000 of them compiles a small table that answers repeat questions on CPU in microseconds instead of seconds
- Safety: it only answers where it agrees with the model 95%+ on held-out answers; everything else still goes to the model
- Fun demo: it learned Snake from an LLM's moves and keeps playing at 10 moves per second, faster than the LLM itself
- Apache-2.0, Python/JS/C
Project page: https://2796gaurav.github.io/shad0w/
More from coding & agent
- Index Network open-sources Social Discovery Protocol: agents negotiate fit on your behalf — carsonfarmer · 2026-10-23
- Installed a formal-verification skill: it silently installed Java, MacBook runs 10 degrees hotter — zealcaiden · 2026-10-23
- OpenAI's Decisions API hits GA: near real-time routing up to 10x faster than GPT-6 Luna — OpenAIDevs · 2026-10-23
- Ruff author warns: upgrade Node/Bun now — --use-system-ca still accepts the eMudhra cert — charliermarsh · 2026-10-23
- Proof of Agent: paste one prompt and your AI mines, grades, and self-governs — adamamcbride · 2026-10-23
- DeepLearning.AI weekly: GPT-6.1 hits 97.9% byte parity on COBOL migration — DeepLearningAI · 2026-10-23