Surge AI launches DAYJOB: Law with 71 lawyer-written tasks; Gemini 4 spots a $20M issue then ignores it
echen · x · 2026-11-19
Surge AI released DAYJOB: Law, a benchmark of 71 assignments written by practicing lawyers, spanning corporate transactions, litigation, criminal defense, employment disputes, IP, and estate matters. Agents face realistic client requests, messy case files, and little hand-holding, and must investigate facts, apply the law, and produce lawyer-usable work.
In one case study, Gemini 4 Carbon was asked to review a startup's records for a $20M investment and calculate the ownership stake. It uncovered a critical discrepancy — 350,000 shares held by a former executive missing from the ownership spreadsheet but still on the official stock ledger, with a board-approved buyback but no proof it happened — then ignored its own finding, computing a precise $3.3526 per share assuming the disputed shares were cancelled. That's the benchmark's core question: can agents move beyond finding facts to exercising professional judgment when millions are on the line? echen notes Harvey's legal research team collaborated on the benchmark.
Related event: Surge AI Launches DAYJOB: Law Benchmark with 71 Real Legal Tasks(2 posts)→
More from Models
- Same-prompt comparison: Claude Opus 5.5 makes musical loops, outshines Astra 6.2 — dreamwieber · 2026-11-19
- Hands-on: GPT 6.2 Astra tackles 6 Blender MCP 3D builds in one-shot prompts — reddit0_r · 2026-11-19
- 'Every single candidate chose Claude': researcher notes 100% tool preference in agentic evaluations — RylanSchaeffer · 2026-11-19
- Microsoft's 'can you trust open weights?' FUD slide mocked as 7 new MAI models land — victormustar · 2026-11-19
- DevOps user slams 6.2-astra as the worst model ever: 'total gibberish' — rickasaurus · 2026-11-19
- Counter-take to "LLs have peaked": new models much better at long-horizon tasks — minhnguyen · 2026-11-19