Harvey shares how it evals contract review when ten lawyers redline ten different ways
graceisford · x · 2026-10-07
Harvey engineer Aveek Dasgupta shared how the team designs evals for contract review agents, with Ryan Daniels noting this tackles a major unsolved transformer problem.
The core challenge: ten great lawyers redline the same contract ten different ways, so there's no single verifiable reward signal for hillclimbing. Legal reasoning is described as the most complex of such multi-answer domains. The team says they've cracked key parts of the puzzle and are now publishing their eval methodology.
Related event: Harvey Team Shares How to Evaluate Contract-Review Agents(2 posts)→
More from coding & agent
- Exa Adds 211M Restaurants, Museums and Parks to Its Agent Search Index — yoimnotkesku · 2026-10-07
- Figma Agent Officially Exits Beta — zan2434 · 2026-10-07
- Building an AI Sales Coach With Hermes: A Voice Buyer That Pushes Back and Scores Every Call — tomcrawshaw01 · 2026-10-07
- Ampersand Builds Integration Infrastructure Powering Enterprise Agents in Salesforce, SAP and Workday — hey_abusiddik · 2026-10-07
- Garry Tan Praises Capy's Cross-Thread Coordination as SOTA in His Parallel Coding Workflow — garrytan · 2026-10-07
- CUAWright: Terminal-Only Computer-Use Agent Beats GUI Harnesses, Cuts Cost 37.5% — ysu_nlp · 2026-10-07