Grok 4.6 lands #2 on new DiligenceBench, statistically tied with Claude Opus 5
karinanguyen · x · 2026-08-19
DiligenceBench is a new equity-research agent eval jointly released by Paper Instruments and Thoughtful Lab: 150 open-ended diligence tasks across 10 sectors and 143 companies, with 5,428 rubric criteria measuring factual accuracy, analytical reasoning, and risk awareness. Tasks require external research (SEC filings) rather than recall, and the benchmark evaluates the full agent system—model, retrieval, tools, and instructions—across three harness tiers (H1 Loop / H2 Sandbox / H3 Finance).
Grok 4.6 enters the Finance harness leaderboard at #2 with 53.2, 1.1 points above Claude Opus 5 (52.1); the paired 95% CI of −1.0 to +3.2 means the two are statistically tied.
Behavioral differences stand out: Grok averaged 41 tool calls per task vs. 22 for Opus, and made 3,293 SEC filing searches vs. 486—broader search helped it find more evidence and follow task instructions better. Author Karina Nguyen argues the benchmark already captures some inference-time scaling (H1→H3), but more evals are needed to understand how performance scales with compute, search, and tool use.
Related event: Grok 4.6 Ties for Second on DiligenceBench at Low Cost(2 posts)→
More from Models
- Prediction: frontier AI revenue growth could slow even as AI usage keeps rising — yi111 · 2026-10-16
- Meta teased 816B-param open-weight Muse Spark Open with 1M context, says meme thread — alexandr_wang · 2026-10-16
- Nolla Health Post-Trains Nemotron 3.5 VL: Visual Diagnosis Jumps from 35% to 66.8% — LoganGrasby · 2026-10-16
- Dev argues coding models now beat nearly all engineers in every branch of software — zack_overflow · 2026-10-16
- Prediction: Chinese frontier models could match GPT-6 by March 2027 — teortaxesTex · 2026-10-16
- Liquid AI's d1-ultra, #1 on Artificial Analysis Decision Index, lands on Vercel — JosephJacks_ · 2026-10-16