Legal Tasks: Models Meet 90% of Criteria but Fail Full Pass
ArtificialAnlys · x · 2026-07-08
Artificial Analysis found that models can handle most requirements of a legal task but rarely satisfy all of them. Leading models can achieve over 90% on individual scoring rubrics, yet their full pass rate for entire tasks sits between 0% and 14.2%. Only 4 models exceeded a 90% rubric pass rate: Fable 5 (93.6%, the highest), Opus 4.8 (91.1%), GLM-5.2 (91.0%), and Sonnet 5 (90.1%).
Related event: Artificial Analysis Releases Harvey Legal Agent Benchmark Results(8 posts)→
More from Models
- Sam Altman is headed to Washington to brief Congress on OpenAI’s GPT-6 line — inductionheads · 2026-07-22
- Benchmark chart pits GPT-5.6 Luna, Grok 4.5 and Gemini 3.6 Flash on price and scores — iruletheworldmo · 2026-07-22
- Claim says Kimi was distilled from Fable, sparking a model-attribution jab — cephaloform · 2026-07-22
- Gemini 3.6 Flash is now available in Antigravity and chat — MartianOnJupiter · 2026-07-22
- Gary Marcus says LLM math skills are like knowing only a car’s engine size — GaryMarcus · 2026-07-22
- OpenAI’s Codex + GPT-5.6 Sol hits 99% recall in Project APE verification tests — soumitrashukla9 · 2026-07-22