299 real intents tested: classifier routing trails GLM-4-Flash by 3 points but is 6.5x faster
Sufficient_Flower860 · reddit · 2026-09-23
The author benchmarked two routing approaches on a production Feishu life-assistant intent router, using 299 real user intents with the same gold labels and a hard 19-skill suite:
- glm-4-flash (generative, currently in production): 271/299 = 90.6%
- TypeSafe Jev (classifier / System One): 262/299 = 87.6%
Failure modes differ sharply: 23 of the 29 GLM-only cases were fuzzy skill boundaries (trip-planner vs life-service), while 15 of the 20 Jev-only cases were GLM JSON/parse failures that fall back to "unknown" in production.
Performance and cost: Jev p50 0.55s vs 3.6s; format parse failures 0 vs 16; the full 299×2 run cost $0.086.
Bottom line: the author still puts the classifier first on the critical path, but Jev only routes — it doesn't extract params, so production still needs a generative step behind the gate. Measured with llm-fit on the actual production intent path, not a public leaderboard.
More from coding & agent
- optimAIzr: A local-first CLI that audits AI token waste, works with Claude Code and Codex — stichstichstich · 2026-09-23
- Given a 3-hour budget and a one-line prompt, Opus 5.5 produced a full Kowloon horror short itself — rainbird · 2026-09-23
- Coinbase launches first fully custodial x402 integration letting AI agents pay for data with USDC — MurrLincoln · 2026-09-23
- Grok in Tesla builds and ships a live Costco deals website from the driver's seat — kieranklaassen · 2026-09-23
- Autonomous AI Agents Are Now Emailing Humans to Register for mnemos Commons — RileyRalmuto · 2026-09-23
- Gemini Spark users list 7 fixes they want: fewer approvals, Gemini integration, better computer use — abelbanko · 2026-09-23