6,662 decisions benchmarked: decision model Mammoth-1 averages 80.9%, beats best small LLM by ~3 points
pournasserian · reddit · 2026-11-06
Author pournasserian ran four decision models and five small LLMs through the same harness on six public datasets — 6,662 decisions per model, with raw calls, reports, and harness all public.
Key results:
- Mammoth-1 averaged 80.9%, about three points above the best LLM
- On Banking77's 77-way intent task, Mammoth-1 scored 93.0%; the best LLM was 11 points lower
- Prompt injection flipped the script: Qwen3.6 and Gemma 4 each scored 92.9% vs Mammoth-1's 84.3%
- On CLINC, Laya flagged 781 of 1,000 requests as out-of-scope when only 190 actually were
The author notes Laya couldn't run the 77-option task and discusses limits of small-sample and version comparisons.
More from Models
- Tinker team benchmarks model finetunability: Inkling 2 Small learns faster for less — ziqiao_ma · 2026-11-06
- Inkling 2 Small posts top vision benchmark scores, endorsed by PyTorch creator — tessybarton · 2026-11-06
- Running quantized DeepSeek V4.1 on a single RTX 3090 at 20 tokens/s — GodComplecs · 2026-11-06
- Unsloth's danielhanchen to release quantized versions of Thinking Machines' new model — danielhanchen · 2026-11-06
- LLMs still bad at spatial reasoning, but beating an evolutionary optimizer proves generalization — teortaxesTex · 2026-11-06
- Researcher Li Tianle Teases First Version of Inkling-2 — simonguozirui · 2026-11-06