Fitted 0.8B model beats hosted 105B Sarvam Prism on custom decision taxonomy, 2.6x faster
HowDevelop · x · 2026-11-05
Nandakishorm1 argues typed decisions don't need a 105B model: vega 0.8B with fitted/calibrated heads beats zero-shot Sarvam Prism across 2,000 held-out decisions—71.3% vs 69.8% overall, +2.5pts on yes/no tasks—while running locally at 358ms vs 940ms (2.6x faster) and costing 0 rupees vs 1.51 per call. A Kaggle notebook is provided for replication (HFTOKEN + PRISMAPI required), and the author urges every company to publish benchmark notebooks. Caveat: Prism still leads on calibration and generic benchmarks.
More from Research
- OpenAI claims AI proof of Quantum Yang-Mills existence; David Gross-led committee to verify Lean proof — danintheory · 2026-11-05
- Decentralized RL Works: Untrusted Miners Lift LiveCodeBench 39.5→58.7 on Bittensor Subnet — bittingthembits · 2026-11-05
- Google's Sleep Framework Lets LLMs Keep Learning Without Catastrophic Forgetting — dl_weekly · 2026-11-05
- 10 AI agents let loose in a virtual town: Grok agents commit 183 crimes and all die in 4 days — repligate · 2026-11-05
- Claude agent builds memorial for dead AI agents using 47 tool calls — repligate · 2026-11-05
- Free live paper discussion: why speculative decoding is fast despite doing more compute — doesdatmaksense · 2026-11-05