DecisionPivot: 40k-item benchmark shows item accuracy hides LLM decision-model gaps
RexDouglass · x · 2026-10-26
HuggingPapers highlights DecisionPivot, a diagnostic benchmark of 48 tasks and 40,033 items showing that item accuracy alone hides large differences between LLM decision models.
- Design: contrast pairs — two traces differing only in step order, testing whether models track which goal conditions are actually met.
- Key results: on public test sets 27B decision models cluster at 52–59, but DecisionPivot pair accuracy ranges 25–87; coverage at 5% error ranges 14–94.
- Perplexity Decider v1.1 answers yes at 1.00 probability to both traces in a pair, missing the failing goal; SL-27B AutoThink leads (87 pair accuracy, 94 coverage).
Takeaway: evaluating agent decision models requires pairwise consistency and coverage-at-error metrics, not raw item accuracy.
More from coding & agent
- A Claude Code prompt that strips ads and tracking off your Smart TV in 15 minutes — alex_verem · 2026-10-26
- Blogger says a Claude Code prompt can strip smart TV tracking and ads in 15 minutes — alex_verem · 2026-10-26
- SpaceX AI engineer turned missing context into skills: 1,000 PRs in a month — nikola_mr64990 · 2026-10-26
- Open-source sky camera run by Claude Code catches 17 meteors overnight, cuts star false alarms to 7% — Terrible_Award_2124 · 2026-10-26
- AI-built OpenSearch clone validated by 6,000 black-box tests: Rust at 64.5%, Go at 32.6% — Strong-Mousse-1020 · 2026-10-26
- Fully local open-weight pipeline produces 10-min animated episodes at ~4 GPU-hours each — Zangetsu__0 · 2026-10-26