100M+ Tokens: First Fully Simulated Law Firm to Benchmark Legal Agents
marcbhargava · x · 2026-08-08
Engram Lab, in collaboration with legal AI company Harvey, has built Calderwood & Harkness, a fully simulated synthetic law firm designed to evaluate legal agents more realistically.
- Environment scale: Features over 100 million tokens of documents and 250 client cases.
- Core problem: While current AI models know the law, they don't understand law firm operations. Existing benchmarks start tasks from scratch, failing to simulate how lawyers accumulate experience over time.
- Solution: This environment allows legal agents to perform multiple tasks within the same context, enabling them to build on accumulated knowledge just like real lawyers.
Related event: Harvey Open-Sources 100M+ Token Synthetic Law Firm Dataset(3 posts)→
More from coding & agent
- Expert Warns: Avoid Vibe Coding for Infrastructure Software — springrod · 2026-08-08
- DeepSeek Cascade Beats GPT-5.6 Luna on DeepSWE at 37% Lower Cost — togethercompute · 2026-08-08
- Guardrails Hinder Defense: Dev Calls for Open Models to Harden Security — max_paperclips · 2026-08-08
- Can MiniMax H3 Handle Automated Video Generation Pipelines? — datavyro · 2026-08-08
- Open-Source Agent 'sol-advisor' Upgrades with Support for Cursor, Copilot and More — daniel_mac8 · 2026-08-08
- Parahelp Launches Automation: Auto-Sync AI Support Agents with Codebase Updates — Scobleizer · 2026-08-08