ValsAI releases ClaimBench, evaluating 15 models on 1,960 financial claims
denisyarats · x · 2026-10-21
ValsAI released ClaimBench, a benchmark for decision models testing whether a financial document supports, contradicts, or doesn't address a claim.
- 15 models evaluated on 1,960 claims paired with 10-K excerpts.
- Decider V1.2 leads the decision-model pack; the Pareto frontier includes Mercury Decide, d1, and budget option Decider Flash V1.2 (96.8% accuracy at lowest cost), with general models like Claude Haiku 5.5 and Claude Opus 5.5 as comparisons.
- Charts plot accuracy against per-claim cost ($0.0001–$0.1, log scale).
Related event: Perplexity's Decider V1.2 Tops Vals AI's New ClaimBench(3 posts)→
More from Models
- DeepSeek V4 called 'god-tier infra, 4th-tier ML': 7 tricks stacked, 20 patches to fix them — teortaxesTex · 2026-10-21
- GPT-6 Astra and Claude Fable 5.1 write $10,000 research papers — experts reject all — sayashk · 2026-10-21
- GitHub Copilot's HydraFusion Orchestrates Multiple Models Per Task — WirelessLife · 2026-10-21
- OpenVLHarness turns any MLLM into a visual reasoner, +9.9 points across 22 datasets — yining_hong · 2026-10-21
- User claims Gemini analyzed his tennis match video and improved his game — zacharynado · 2026-10-21
- Opus 5.5 tops OpenRouter spending in 20 of 29 task categories — mmurph · 2026-10-21