Benchmarking System 1 decision models: 40 yes/no questions in one pass, 159ms agent routing
ThatOneSnare · reddit · 2026-11-03
A Reddit developer of open-source agent harnesses (Herald, Ultimate Assistant) benchmarked System 1 decision endpoints (tev1, Jev, pplx-decider-v1.2, Liquid d1.1) against a 9B generative LLM for agent routing and tool gating.
Key results:
- The core trick: instead of summarizing a prompt or asking one routing question, ask 40 orthogonal yes/no questions (5 specialist, 25 sub-intent facets, 5 conversational exclusions, 5 commitment gates) in a single forward pass.
- Hosted APIs: all 40 questions evaluate in 159-319ms at $0.00003-$0.00006/turn with zero output tokens; the 25 sub-intent facets break routing ties in plain Python, cutting LLM fallbacks from 15% to 0/24 with 100% accuracy.
- Privacy bonus: structured semantic briefs can be reconstructed purely from boolean probabilities with fixed facet codes, never logging raw prompts.
- Local single-GPU (tev1 4.3B Q80 on RTX 4070 Ti SUPER): Ollama evaluates in waves (416ms for 40Q), so the router switches to a lean 5+1 split — 27ms per gate, 100% accuracy (56/56) on unseen routing tests, versus 93% and 477ms for qwen3.5:9b with lexical rules.
More from coding & agent
- Unofficial Crunchyroll MCP server ships 19 tools with browser-mediated sign-in — Pleasant-Today-6584 · 2026-11-03
- EAT Evolves Agent Teams to Hit SOTA on Kernel Optimization, Ranks #4 in Open Kernel Challenge — berkeley_ai · 2026-11-03
- Alchemy open-sources an agent-friendly test runner built to replace vitest — samgoodwin89 · 2026-11-03
- Alchemy adds keyless AWS↔GCP auth via OIDC Identity Federation, drops service account keys — samgoodwin89 · 2026-11-03
- Alchemy 2.0 beta ships Effect 4.1 support, DynamoDB state store with ~200x speedup, and 95+ fixes — samgoodwin89 · 2026-11-03
- Alchemy Domain ships event replay, orchestrating millions of Durable Objects via fan-out — samgoodwin89 · 2026-11-03