Will-It-Jev: Rust 5-tier cascade router routes simple prompts away from LLMs with sub-5ms p99 latency
forestmars · x · 2026-10-08
Will-It-Jev is an open-source Rust engine built on a simple premise: stop sending simple unit conversions and structural lookups to expensive generative LLMs.
It implements a five-tier deterministic fallback cascade that intercepts intent before token generation even starts, using exact/fuzzy pattern scanning to triage traffic between discriminative (Jev) decision heads and generative LLM pathways. The project claims p99 decision latency under 5ms for the overwhelming majority of traffic, targeting ultra-low-latency, high-throughput routing.
More from coding & agent
- Evidence launches: image search for AI agents retrieves source-backed charts in 125ms — RichardsonDx · 2026-10-08
- The ideal WordPress AI agent finds the conflicting CSS rule instead of stacking another !important — HaktanSuren · 2026-10-08
- The agent-first app checklist: seven surfaces from WebMCP to ChatGPT plugin, buildable in a day — seanwbren · 2026-10-08
- Researcher's agent workflow: label 100 examples, have the agent scale to 10k pseudo-labels — ducha_aiki · 2026-10-08
- One user's daily agent stack: 7 AI agents split by task, with OpenAI's Dots starting a business — omooretweets · 2026-10-08
- Dev shares his bot setup: one bot per project plus a primary Grok bot for everything else — jonathan_wilke · 2026-10-08