System 1 models, not LLMs, should handle routing, gating and scoring in AI systems
arpit_bhayani · x · 2026-10-31
Arpit Bhayani lays out a framework for splitting decision-making in AI systems: LLMs are autoregressive — one forward pass per token — great for reasoning and generation, but most subsystem decisions (routing a request, gating an action, scoring a candidate, classifying input) just need direct prediction, not token-by-token generation.
Core rule
- Direct prediction (label/score/route/yes-no) → System 1 model: high-volume, low-latency
- Multi-step reasoning or open-ended generation → System 2 (the LLM)
System 1 model families
- Reward models for scoring
- Classifiers for labels and routing
- Non-autoregressive typed decisions
- Extraction models for structured fields
Takeaway: System 1 models become the decision infrastructure for routing, gating and scoring, so the LLM doesn't have to be the decision engine for everything inside your AI system.
More from coding & agent
- 174-line mental schema + Claude Code: an agent workflow that builds entire slide decks — tianshi_li · 2026-11-01
- Stripe Link goes live in Agent Picnic to manage AI agent connections and permissions — jeff_weinstein · 2026-11-01
- BossBot books pickleball courts at midnight daily, beating the reservation rush — chrisfleck · 2026-11-01
- OpenAI's Astra 5.5 runs massive code patches just to list Devcon side events — devanshmehta · 2026-11-01
- Curated skills collection lets you run agents on your own harness and subscriptions — jasonkneen · 2026-11-01
- Workshop: Build a media agent on CPU, call image and TTS models as MCP tools — Al_Grigor · 2026-11-01