Designing evals for contract review agents: ten lawyers, ten redlines
graceisford · x · 2026-10-07
In a thread shared by Grace Isford, Aveek Datta argues that designing evals for contract-review agents is among the hardest technical problems he has faced: ten great lawyers redline the same contract ten different ways, making it hard to specify what 'good' looks like for an agent. His team says it has cracked major parts of the puzzle and published a detailed write-up.
More from coding & agent
- Trying to Plug Open-Source Mistral Into an Agentic Coder Just Doesn't Work, Says Berman — MatthewBerman · 2026-10-07
- Meme: Your Agent After Its 50th Context Compaction — dejavucoder · 2026-10-07
- DJ app's MCP server lets Claude drive real synths and drums instead of generating audio — tech_sand · 2026-10-07
- Dev builds research agent with mandatory citations and an LLM judge that strips weak ones — FreakFrakFrok · 2026-10-07
- 20 open-source OCR & PDF extraction tools for RAG pipelines, sorted into 5 categories — MaryamMiradi · 2026-10-07
- Anthropic designer interview: three designer archetypes and early Claude Code lore — Flomerboy · 2026-10-07