Self-hosted 14B hits 88% on contract review vs Claude's 90%, at 1/27 the cost
Broad_Breakfast_1172 · reddit · 2026-10-20
A developer built a private contract-review feature on a self-hosted Qwen 2.5 14B AWQ (one RTX 5090) and benchmarked it against Claude Sonnet 5.5 on lawyer-labelled test sets. Code is MIT-licensed.
Architecture: documents split into 10k-token overlapping parts → OpenAI-compatible endpoint returns strict JSON with an answer plus a quote per field → every quote is verified against the original text; unverified answers don't count — the single most useful design decision.
Numbers
- Commercial contracts (CUAD, 102): 88.0% vs 90.4% hosted, $2.30 vs $63 per 1,000 contracts
- NDAs (ContractNLI, 123): 67.1% vs 86.3%, $1.20 vs $15 per 1,000; per-question reading rules lifted it to 71.8%
- Throughput: 3–4s per NDA, 4 documents concurrent on one GPU
Lessons
- Small models paraphrase evidence under extra instructions (150 answers caught by quote verification); the hosted model never did
- Prompt rules are blunt: +4.7 points overall but one question dropped from 81% to 59%
- 4-bit AWQ within a point of full precision at 1/3 the cost; "OpenAI-compatible" endpoints vary (temperature=0 and jsonobject not universally supported)
- Locating clauses is nearly solved by a 14B; interpreting exceptions is where big models still win
More from coding & agent
- Hamel Husain Benchmarks Six Decision Models for Evals: Threshold Tuning Beats Model Choice — Hamel Husain · 2026-10-20
- Microsoft to let users upload Copilot agents with actions and MCP servers by default on Oct 25 — luisdans · 2026-10-20
- Voyager launches as an 'open harness' driving Claude, Resolve and Blender for filmmaking — ravisparikh · 2026-10-20
- Dev builds a Chrome extension to improve X, starting with a better @ mention feature — vista8 · 2026-10-20
- Grok Bot books a hotel end-to-end, asks approval before paying and flags a duplicate booking — XFreeze · 2026-10-20
- Developer rants Codex session bloat kills usage quota, switches to Hermes-Agent for context control — alexcovo_eth · 2026-10-20