Artificial Analysis Independently Reproduces Harvey Legal Eval
ArtificialAnlys · x · 2026-07-08
Artificial Analysis released LAB-AA, an independent reproduction of the Harvey evaluation. Key differences from the original include: models run on their in-house Stirrup agent harness, which supports context compression instead of failing at the limit; it uses simplified custom agents and judge prompts; it excludes Harvey's custom tools and document generation scripts (like pptx, docx), replacing them with simple code execution tools to reflect raw model capabilities; and deliverables must strictly match specified filenames rather than using fuzzy matching.
Related event: Artificial Analysis Releases Harvey Legal Agent Benchmark Results(8 posts)→
More from coding & agent
- Kolab Benchmark Uses Pro Screen Recordings to Show AI Agents Overestimate Collaboration Quality — minnesotanlp · 2026-10-27
- Dev argues Claude burns far more tokens than GPT models for the same coding work — haider1 · 2026-10-27
- Stop building agents that can't change models: stable contracts, external state, real evals — rseroter · 2026-10-27
- Cheap models can cost more in agent harnesses than frontier ones, researcher warns — vasuman · 2026-10-27
- Kanzo: A 5k-Line Rust Kanban Binary With MCP for Claude Code and Codex — allmudi · 2026-10-27
- 'A human approves everything' is a trap: approval fatigue and how to actually design safe agents — Efistoffeles · 2026-10-27