2026-08-27
On-prem Llama-3-8B extracted 14 fields from 130,509 notes on 8,366 nephrectomy patients at 97.5% agreement; arbitration favored the LLM on 10 of 14 fields.
The reference standard for a clinical research database is still a person reading the chart. Published estimates put that work at about 30 minutes per record or more. Fatigue produces errors, and inter-rater disagreement has been reported at 10%, 26%, and as high as 42%. The process does not scale.
Closed models can extract fields, but notes are protected health information and cannot leave the hospital. The AIM-HI Lab at Cleveland Clinic's Glickman Urological Institute, with IBM Research Israel, tested a narrower claim: an open 8B model, run on-prem, with note-type prompts and branching logic, can rebuild a usable multi-domain kidney-cancer database from real records.
They chose Llama-3-8B as a trade-off between quality and local hardware, downloaded it from Hugging Face, and ran it in FP16 on NVIDIA A100 GPUs with batching where possible. Notes came from Epic Clarity. Inclusion used CPT codes for partial or radical nephrectomy in the health system from October 2009 through July 2024; nephroureterectomy for upper-tract urothelial carcinoma was kept under the same definition. IRB 23-1158 classified the work as minimal risk and waived consent.
The pipeline has four stages:
Missing manual values were excluded from comparison. Categorical fields used percent agreement and Cohen's κ; continuous fields used ICC. Blinded third-party raters adjudicated a random 50 mismatches per variable (or all mismatches if fewer than 50).
The model processed 9,095 patients; 8,366 overlapped the manual reference. Mean age 62.8 years, 63% male, 84% White, 57.1% partial nephrectomy, 51.2% clear-cell RCC. From 130,509 notes (15.6 per patient) the pipeline produced 136,425 fields. Overall agreement was 97.5%, with per-field values from 74.2% to 99.5%.
| Field | Agreement | κ or ICC | Note type |
| Rhabdoid features | 99.46% | κ=0.94 | pathology |
| Solitary kidney | 99.01% | κ=0.36 | radiology |
| Sarcomatoid features | 98.82% | κ=0.91 | pathology |
| Histologic subtype | 98.76% | κ=0.94 | pathology |
| pT stage | 98.58% | κ=0.95 | pathology |
| Grade | 98.03% | κ=0.95 | pathology |
| Pathologic size | 89.15% | ICC=0.92 | pathology |
| Ischemia type (cold / warm) | 74.67% | κ=0.65 | operative |
Most pathology fields cleared 98% agreement and κ above 0.90. The weak spots were ischemia type in operative notes and pathologic size. Solitary kidney shows the κ trap: 99.01% agreement but κ=0.36, because the event rate was under 0.1% and chance agreement is already high. Surgical margin status did the same (93.09%, κ=0.55). Estimated blood loss sat at 91.61% agreement with ICC=0.71.
Agreement rose from 84.2% in 2009 to 97.8% in 2023, which the authors tie to more standardized templates.
On the 2.5% of fields that disagreed, arbitration favored the LLM for 10 of 14 variables. Manual extraction won more often on pathologic size, pN, sarcomatoid features, and ischemia type. A typical size error: the report lacked three dimensions, so the model grabbed a fat-over-tumor measurement. A typical pN error: when the report omitted the stage, the model wrote pNX instead of pN0.
Median note length was 2,722 characters; mean compute was 14.5 seconds per note. Operative notes came from 162 surgeons across six hospitals in more than one state. Seven A100s finished the cohort in 82.1 hours, about 35 seconds per chart. Recreating the same set by hand at 30 minutes per chart would take one full-time worker about two years.
Among published LLM chart-review studies with manual validation, this is a large one on both patient count and field breadth. The result is not "GPT-4 can read notes." An 8B open model, kept on-prem, with branching prompts, matched a manual kidney-cancer database across hospitals, a 15-year window, and more than a hundred authors, and it won most of the disputed fields. For a hospital that cannot send PHI to a vendor API, that path is more usable than calling a closed model.
The cost is also clear. Clinicians iterated the prompts. A new disease or a new hospital will need its own validation. The authors credit the scores first to prompt iteration and branching, not to model size.
Single health system, surgical cohort, no accuracy split by site or region. Longitudinal outcomes (recurrence, progression, treatment response) were not extracted. Llama-3-8B is small; whether a larger or clinical-domain model would do better is untested. Arbitration covered a random subset of mismatches and was not done in duplicate, so the gold-standard question is only partly closed. The manual database itself has substantial missingness; agreement is computed only where both sides have a value.
The title talks about the beginning of the end for manual chart review. The evidence supports a narrower sentence: on this nephrectomy database, a local 8B model can replace most baseline field abstraction. It does not replace the clinicians who write prompts, and it has not yet been asked to read follow-up notes, which are messier.