5ms Router Slashes LLM Costs by 96%
Negaaaa7 · reddit · 2026-07-19
To cut LLM API costs, the author built a 0.57 MB router that decides between local and cloud models, boasting an inference time of roughly 5 ms without needing embedding models or vector databases.
The router simultaneously handles three tasks:
- Local/cloud routing
- PII detection
- Jailbreak / injection identification
They share their test results, including an offline evaluation of 2,022 real prompts, jailbreak detection rates, false positive rates, and known failure modes (e.g., certain "act as..." role prompts easily trigger false positives).
It's currently available in three formats: a VS Code/Cursor extension, an npm SDK, and a hosted API. The author emphasizes that the extension is merely a frontend, while the actual local classifier runs within the SDK.
Related event: Tiny 0.57MB Router Slashes LLM Costs(2 posts)→
More from coding & agent
- 250 Builders Hack With Codex at Berlin Community Hackathon Ahead of OpenAI DevDay — paw_lean · 2026-10-26
- Ben Bajarin: Frontier AI is becoming the enterprise cyber-defense stack — BenBajarin · 2026-10-26
- Building AI bots on raw APIs? Devs say models are 'dumb without the harness' — madeofAjala · 2026-10-26
- HarnessMM cuts agent tokens per task by up to 88% while raising success rates — omarsar0 · 2026-10-26
- Microsoft demos intent-classifying agents built with Copilot App and Foundry, deployable across Teams and Outlook — marlene_zw · 2026-10-26
- Looking for a wearable mic with best battery life that feeds recordings to personal agents via MCP — altryne · 2026-10-26