Skip the LLM for routing? Jev classifier ties Llama 70B in harness tests
2BucChuck · reddit · 2026-09-21
A developer shared tests on choosing a mode classifier for their agent harness (switching local models between CHAT/RESEARCH/PLAN modes, like Claude Code) on Reddit.
- Baseline: Llama 3.3 70B was the best classifier to date; Jev tied it in the new tests.
- Key insight: what matters isn't speed or accuracy but trustworthy probabilities — an LLM only outputs a guessed probability, so you can't tell when uncertainty enters your pipeline. When Jev reported 90% confidence, it matched the judge model Fable nearly 100% of the time.
- Method: multiple models graded by Fable on 50% synthetic + 50% real log data; Claude models ran on Bedrock so latency isn't apples-to-apples.
- Surprise: Qwen Next matched Fable 85% of the time in raw comparison, far above the agreement rate of others.
Related event: Jev Decision Model Matches LLMs at 10x Speed for Routing Tasks(2 posts)→
More from coding & agent
- GitHub Copilot App uninstall leaves MCPs, plugins and old versions behind — PaulShellDev · 2026-09-21
- TypeSafe's Jev Cut Grok Build Costs 22-40% on the Same Tasks — Daniel_Farinax · 2026-09-21
- jev-skill-suggester: a lightweight pre-execution skill router stops agents picking wrong tools — udmrzn · 2026-09-21
- OpenClaw adds FaceTime: your agent can now call you, and answer your calls — steipete · 2026-09-21
- Reverse-engineering Instinct's memory: beautifully simple, replicable in 60 lines — julianweisser · 2026-09-21
- 3 Weeks of Logs Show 40% of Messages to Coding Agents Are Overhead — Lucky-Group9525 · 2026-09-21