typed-decisions is now an official HF benchmark ranking 2,274 open-source decision models
LysandreJik · x · 2026-10-20
Hugging Face announces that the LocalLLaMA community's typed-decisions dataset is now an official benchmark on the Hub, with a live leaderboard filterable by parameter size from <1B to <500B. It answers which of the 2,274 open-source decision models on the Hub you should actually use.
- Task is text classification for structured decisions, with calibration and probabilistic-classification tags; 3,200 rows, mostly synthetic, Apache-2.0 licensed.
- Current leaders: OPENGRAPE/Jeva-9b at 0.82, Darwin-27B-ZTC-v5 at 0.81, and a 4-bit quantized gemma-4-e2b-it at 0.80. A 150M-parameter model scores 0.79, beating Perplexity's official pplx-decider-v1.1-27b (0.75) and the vllm-sr/d3 family (0.75).
- Quantized small models consistently approach larger ones, suggesting limited scale returns on structured decision tasks.
More from Research
- New PhD dissertation finds software engineers rarely blow the whistle on ethical lapses — guzdial · 2026-10-20
- 361-page paper details 151 trading strategies used by hedge funds, free PDF — mdancho84 · 2026-10-20
- Conditional Program Shrinks LLM Lookup Tables 75% While Improving Fidelity — dnotitia · 2026-10-20
- Ablation study: domain-scoped editing worth up to 3.7 points in HARNESS² agent framework — morgymcg · 2026-10-20
- Indie researcher stores documents as SSD "memory cartridges" for a frozen Qwen2.5-72B, claims byte-identical replay — Nearby_Indication474 · 2026-10-20
- Survey compares Vision Encoders across 9 frontier models, from SigLIP 2 to encoder-free — liuziwei7 · 2026-10-20