A 0.57 MB router model cuts LLM costs by 68%
Negaaaa7 · reddit · 2026-07-20
The author claims they cut LLM costs by 68% with Prompt Compass, a tiny 0.57 MB model that handles four classifications in a single CPU call of about 5 ms.
What it does
It combines four tasks that are often split across multiple models and network hops:
- prompt routing
- PII scanning
- jailbreak detection
- local/cloud lane selection
Reported results
- 2,022 held-out real prompts
- 82% overall accuracy on the hard 4-way task
- 87.5% jailbreak recall on lmsys/toxic-chat
- under 5% false blocks on genuine prompts
- 100% recall on 15 obfuscated PII attacks in their benchmark comparison
Where it fails
- about 1 in 5 prompts are misrouted, mostly from local to cloud, which hurts cost rather than safety
- persona prompts like “act as...” often hit the jailbreak lane and are flagged for review
- it does not catch slow-burn multi-turn attacks because it only sees one prompt at a time
Integration
The tool ships as:
- an on-device npm SDK
- a hosted API
- a VS Code / Cursor extension
The extension is a thin client to the hosted API, while the SDK runs locally. It cannot intercept built-in Copilot/Cursor chat. The extension is also available on Open VSX and has a free tier.
Related event: Tiny 0.57MB Router Slashes LLM Costs(2 posts)→
More from coding & agent
- Alex Townsend posts 200 open problems in numerical linear algebra for humans and AI agents — IgorCarron · 2026-09-11
- Kimi K2.8 Preview rolls out: near-K3 coding performance, 1M context for all tiers — teortaxesTex · 2026-09-11
- Looking for a classifier of software engineering task shapes to pick models per task — StewartalsopIII · 2026-09-11
- Steal this idea: prompt-to-hardware where agents assemble custom devices — paraschopra · 2026-09-11
- Model Is the Least Interesting Part: A Guide to Six Core AI Architectures from RAG to Multi-Agent — goyalshaliniuk · 2026-09-11
- Non-coder builds layered memory architecture: 20k tokens tracks a year of agent conversations — matteoianni · 2026-09-11