5ms Router Slashes LLM Costs by 96%
Negaaaa7 · reddit · 2026-07-19
To cut LLM API costs, the author built a 0.57 MB router that decides between local and cloud models, boasting an inference time of roughly 5 ms without needing embedding models or vector databases.
The router simultaneously handles three tasks:
- Local/cloud routing
- PII detection
- Jailbreak / injection identification
They share their test results, including an offline evaluation of 2,022 real prompts, jailbreak detection rates, false positive rates, and known failure modes (e.g., certain "act as..." role prompts easily trigger false positives).
It's currently available in three formats: a VS Code/Cursor extension, an npm SDK, and a hosted API. The author emphasizes that the extension is merely a frontend, while the actual local classifier runs within the SDK.
Related event: Tiny 0.57MB Router Slashes LLM Costs(2 posts)→
More from coding & agent
- Dev builds interactive 3D product experience with GPT-6 Astra + Hyper3D Rodin — nikola_mr64990 · 2026-09-11
- Codex tip: use Sol with Astra and Luna sub-agents to save usage — pvncher · 2026-09-11
- agents-best-practices: a provider-neutral Agent Skill for designing and auditing agentic harnesses — tom_doerr · 2026-09-11
- Cognition's SWE-2 uses a KKT duality argument in RL to shift the effort Pareto curve — YouJiacheng · 2026-09-11
- First-ever Three.js Conference lands in Paris, with a panel on AI-shortened design workflows — OdinLovis · 2026-09-11
- Data engineering, not agent frameworks, is the real bottleneck for enterprise AI agents — dhruv2038 · 2026-09-11