4 decision-model leaderboards compared: fine-tuning beats size under 5B, rankings disagree
Balance- · reddit · 2026-11-04
Comparing System One Bench, Artificial Analysis Decision Index, AIM System One Index, and a 100-model Borda leaderboard for self-hosted "decision models": under 5B, the fine-tune matters more than the base (up to 22 points gap on same base; KnowLine 70→80 from Gen2 to Gen5.5 on the same 4B weights). The sweet spot is 4–9B — KnowLine 4B Gen5.5, D3 Mini, and D3 Flash (8B) beat half of 27B fine-tunes. Below 1B, Arabic drops 16–22 points while English holds up; start at 2B+ for multilingual. Calibration survives at small sizes (Decider Flash V1.2 at 1.8% error), but some sub-1B models sit at 10–20%. No single "#1" is trustworthy: two leaderboards correlate at ρ=0.33. GGUF builds get 3–10× the downloads of originals.
More from Models
- Team says models trained across distributed GPUs work well, plans to share one soon — markjeffrey · 2026-11-04
- Researcher seeks help: hidden verifier prompts make his model eval-aware — nrehiew_ · 2026-11-04
- MiniMax H3 Lite and Highspeed 50% off until year-end, exclusive to Replicate and Cloudflare — ritakozlov · 2026-11-04
- Leaked chart shows OpenAI internal model scoring 0.875 on BenchCAD 2.0 vs GPT-6 Astra's 0.594 — ChrisGPT · 2026-11-04
- '6 Astra High' spotted in model picker, hinting at new Astra release — gethackteam · 2026-11-04
- DeepSeek V4.1 Flash reads raw weights: 3 of 4 RL checkpoints identified correctly — poppear · 2026-11-04