Crosby trains judge models to evaluate legal AI where no unit tests exist
graceisford · x · 2026-10-22
- Crosby (an AI legal company) published "Scaling Legal Judgment", arguing that expert work has no unit tests, so eval recipes from math and code don't transfer to non-verifiable domains.
- Their approach: train a model to predict whether an attorney accepts an AI-generated contract redline as written, relying on custom learned judges and pairwise comparisons.
- Motivation: the same suggested contract edit often gets three different outcomes from three attorneys, so evaluation criteria are inherently subjective and demand purpose-built methodology.
Related event: Crosby Trains Legal AI Verifier to Predict Lawyer Acceptance(3 posts)→
More from AGI Musings
- AI industry internalized 'we're not cool' and pivots consumer apps at 35-year-olds — wordgrammer · 2026-10-22
- Peter Diamandis: AI doing 9th-grade homework is a travesty, using it to design spaceships is education — PeterDiamandis · 2026-10-22
- Physics says we are still very far from the limits of technology — burny_tech · 2026-10-22
- 'Zeno's Singularity': the name for gentle, low-slope recursive self-improvement — bradneuberg · 2026-10-22
- Researcher: Race for Speed Systematically Rewards Less Safe Model Development — davidmanheim · 2026-10-22
- Safety researcher: racing dynamics systematically undermine model safety, and it's not inevitable — Hugging Face Blog · 2026-10-22