Decision models match LLMs on safety classification and tool routing, at far lower cost
randal_olson · x · 2026-10-16
Randal Olson digs into Artificial Analysis' new Decision Index, which benchmarks 11 decision models against 15 LLM configurations (with structured outputs) across 4 tasks.
- On the blended overall score, most LLMs outrank every decision model — but that gap comes almost entirely from knowledge questions and hard reasoning, where LLMs win by a wide margin.
- On the two tasks decision models are actually built for — safety classification and tool-call routing — the two groups are roughly even. The best decision model beats every tested LLM on safety classification, and only Claude Opus 5.5 outperforms the best decision models on routing, at 50x+ the cost.
- Every decision model answered faster than every LLM tested on those tasks.
- Takeaway: decision models aren't just worse LLMs; they're a sensible choice for low-cost, low-latency fixed-choice routing, while LLMs remain unrivaled for knowledge and reasoning.
Related event: Decision models match LLMs on routing and safety tasks at 7-8x speed(3 posts)→
More from Models
- Gemini 4 Argon hits AssemblyAI's LLM Gateway, tops voice bench at 99.5 across 50+ models — AssemblyAI · 2026-10-16
- Meta's open-weight Muse Spark Open hits Ollama with 1M-token context and Claude Code support — ollama · 2026-10-16
- 'Opus 5.5' collage demos spark claim: models have taste, no moat — yuwen_lu_ · 2026-10-16
- PrismML ships Bonsai 2 ternary models: 9B fits in 2GB for local AI — gajesh · 2026-10-16
- Apple MLX Runs 816B Muse Spark Open at 36+ Tokens/s on Four Mac Studios — andrejusb · 2026-10-16
- vLLM Ships Day-0 Support for Meta's 816B Open-Weight Muse Spark Open — woosuk_k · 2026-10-16