DecisionPivot benchmark: two 27B models 2.6 points apart differ 70.1% vs 44.2% on contrast groups
startlux-models · hf · 2026-10-21
StartLux releases DecisionPivot, a diagnostic benchmark for decision models (routing, guardrails, output verification) in LLM systems: a core track of 7,800 items across 13 task families with solver-verified contrast groups, plus a 32,233-item reference track. Evaluating 40 decision and 14 general models, it shows aggregate scores hide big gaps: two 27B models 2.6 points apart on the reference track score 70.1% vs 44.2% contrast-group accuracy; Perplexity Decider ranks pairs correctly in 92.3% of cases but answers both members correctly only 74.1% of the time. GLM-5.3 reaches 93.5% contrast-group accuracy at high reasoning effort vs 81.5% at the lowest.
More from Research
- Hamel Husain: BenchGate is pervasive — look at failing traces in any public benchmark — HamelHusain · 2026-10-21
- Menlo Research open-sources Cyclotron, an RL training framework for its Asimov humanoid robot — freelerobot · 2026-10-21
- Father-son duo uses Hark to semi-autonomously medal on ARC Prize after days of self-revision — Deep-Firefighter-279 · 2026-10-21
- Berkeley paper: first tractable estimator of empowerment from offline data for exploration — svlevine · 2026-10-21
- Paper: The network around a cortical core is a regulator, not an optimizer — maier_ak · 2026-10-21
- Big day for AI in bio: OpenAI-Orchestra Bio deal, Holon-S1 cancer model, more — iScienceLuvr · 2026-10-21