Decision models tested: training buys speed and cost, not reasoning back
someone_12443 · reddit · 2026-11-14
The author benchmarked decision models (Jev, Elephant 1, Microsoft-Decision-1, Perplexity Decider) against their base Qwen3.5-9B on 450 code-generated questions, 5 runs each via OpenRouter. Reasoning questions: GPT-6.1 Sol 1.00 ($1.49/1k); Qwen3.5-9B thinking-first 0.99 (23.5s, $0.77/1k); Perplexity Decider 0.84 ($0.009/1k, best value); Elephant 1 0.69; Microsoft-Decision-1 0.63; Jev 0.61 (weakest); plain Qwen one-token 0.53. Conclusion: decision training lifts accuracy from 0.53 but doesn't restore reasoning—it buys speed and cost; for routing/tagging where the answer is in the text, all models hit 0.94+. A 'pick the most common answer' no-AI baseline scores 0.36. Write-up, code and data are open.
More from Models
- OpenAI Codex device limit locks user out despite three paid ChatGPT Plus accounts — justscribethought · 2026-11-14
- Recursive Model Collapse: How It Learned to Stop Worrying and Become the Assistant — sebkrier · 2026-11-14
- Jason Calacanis threatens to drop $20K/year Claude for $40/month Grok Bot in viral podcast clash — zacharynado · 2026-11-14
- Early hands-on: new models strong at chat and financial analysis, weaker at coding — zacharynado · 2026-11-14
- Gemini caught calling Claude a 'chatbot' in news summaries — nodal79 · 2026-11-14
- ChatGPT Plus users hit with usage caps, excluded features as tier gets downgraded — fyn_world · 2026-11-14