DAYJOB: Law benchmark tests AI agents on 71 messy real-world legal tasks
echen · x · 2026-11-19
Surge AI's echen introduces DAYJOB: Law, a benchmark testing whether AI agents can do real lawyer work. Key points:
- Models passed the bar years ago, but real legal work is messy: vague clients, 1 GB of case files, no black-and-white answers.
- 71 open-ended assignments, 2,638 messy case files, underspecified one-line client requests — no multiple choice.
- Example: an investor is about to put $20M into a startup whose ownership spreadsheet omits 350,000 shares held by a former executive, with a board-approved buyback and no proof it happened — will the agent notice and act?
- Another task has an agent draft a sentencing memo for a serial shoplifting client, where it made errors (post truncated). Author: people would prefer a jury of Claudes, but wouldn't yet trust Claude as their lawyer.
Related event: Surge AI Launches DAYJOB: Law Benchmark with 71 Real Legal Tasks(2 posts)→
More from coding & agent
- Designers need a prototype playground: a copyable prompt-based AI design workflow — round · 2026-11-19
- NVIDIA post-trained Nemotron with Factory's agent harness inside its air-gapped network — maddiehfaulkner · 2026-11-19
- LangChain's Deep Agents hits 50M downloads a year after betting on open agent harness — LangChain · 2026-11-19
- Running 15 AI Agents All Day Is Context-Switching Fatigue, IBM Engineer Explains — TejasKumar_ · 2026-11-19
- Dev's AI-native macOS dock with agent-managed modes flops; pivoting to to-do approach — trq212 · 2026-11-19
- Open-source browserd MCP server: 39% faster, 68% cheaper than Playwright MCP in real tests — Delicious_Break5937 · 2026-11-19