Designing evals for AI contract review: ten lawyers, ten different redlines
graceisford · x · 2026-10-09
Crosby Legal founding engineer Aveek Datta explains why designing evals for AI contract review is one of the hardest technical problems: ten great lawyers redline the same contract ten different ways, making it hard to specify what 'good' looks like. Legal is messier than coding — attorney, client, contract and multi-party preferences — with no 'Opus 4.8 moment' yet. Crosby says its approach makes NDA/MSA review 5x faster.
Related event: Crosby Engineer Breaks Down the Hardness of Legal Contract-Review Evals(3 posts)→
More from coding & agent
- Anthropic publishes reference guide for building scheduled automations with Claude Managed Agents — ClaudeDevs · 2026-10-09
- 23 ads, ~100 versions in two weeks: 10 ways AI rewired one studio's video production — justin_hart · 2026-10-09
- Matthew Berman: Prefers Codex as an Interface but Says Opus Is the Better Model — MatthewBerman · 2026-10-09
- MCPJam Engineer: 'MCP Is Bad at X' Conflates Three Very Different Failures — Paoli99 · 2026-10-09
- AI phone agent indie platform now covers numbers in 120 countries, ditching Twilio — redouanea · 2026-10-09
- Building an AI agent 'org chart': multi-agent coordination keeps falling apart — Al_Grigor · 2026-10-09