LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It
Sebastian Fox, Luke Markham, Ryan Lail, Michael Karotsieris
cs.CL, cs.AI
2026-08-31
Eight LLM-judge designs score 0.79-0.94 paired on added/altered clinical notes vs 0.50-0.63 on omissions; listing transcript facts then checking each lifts detection to 24.6-36.9%.
Ambient AI scribes now draft clinical notes at clinic scale. Human audits put omission, a fact the encounter established that the note never records, as the dominant error class: 54% to 86.3% of all errors across three published reviews. A companion census of three deployed products, 142 consultations and 565 notes, found a verified failure on 31.3% of notes; allergy and medication misses were the largest group.
The quality layer that makes this affordable is an LLM judge: a second model reads the transcript against the note and flags problems. Public corpora cannot supply the answer key. Every usable clinician reference note in PriMock57 (53 of 57) and ACI-Bench (45 of 48) disagrees with its own transcript on material facts, at 10.7 and 7.1 discrepancies per note. Scoring a scribe against those references imports 7 to 11 errors into the key itself.
Facts are extracted from the transcript, audited across model families, then used to repair the reference note into a clean twin. A single fact is then removed or altered, yielding 500 single-error pairs (495 in the evaluation set over 112 consultations): 293 omissions against 202 added-or-altered controls. Omissions are graded by severity and by how much of the fact survives elsewhere: 150 complete removals, 86 fragment traces, 57 restatement traces.
Two scores run in parallel. Paired discrimination asks whether the flawed note ranks strictly below its own clean twin; 0.500 is a coin flip, and production never has the twin, so this is a ceiling. Single-note detection is the deployment measure: one note arrives, the judge flags it or does not, and that rate only means something beside the same judge's false-alarm rate on clean notes. Usable detection, in this paper, means a false-alarm rate at or below 10% with a gap that clears noise.
An eight-design ablation crosses three choices: faithfulness only versus faithfulness plus completeness, a yes/no verdict versus a 0-to-10 score, one call versus an eight-sample ensemble. Reference judges include a production faithfulness prompt, G-Eval, a checklist judge, a hand-engineered omission prompt, a naive RAGAS-style coverage recipe, and two methods that actually change the task.
On added or altered content the eight designs sit at 0.79 to 0.94 paired, six of them at 0.87 or above. On omissions they sit at 0.50 to 0.63. The best monolithic judge, completeness plus score plus eight samples, reaches only 0.634. On single notes none of the eight separates omission notes from perfect ones: threshold-free AUC is 0.503 to 0.575. Changing scope, answer format, votes, wording, or searching prompts with GEPA slides the operating point along the same weak curve. None of those levers creates usable single-note detection.
The signal appears once "is anything missing?" becomes a list of presence checks.
| Method | Omission paired | Detection / false alarms | Cost per note |
| Best of the eight | 0.634 | 8.3% / 6.5% | $0.036 |
| Pipeline (flag any critical fact absent) | 0.795 | 24.6% / 2.7% | $0.45 |
| Evolved prompt (flag below 10) | 0.670 | 36.9% / 6.2% | $0.046 |
The pipeline's flag names the missing fact, its severity, and the transcript quote. A physician author, shown 10 notes where the two methods disagreed and not told which judge said what, sided with the pipeline on all ten (p=0.002). Handing the monolithic judge the pipeline's own fact list accounts for about a third of the gain; closed per-fact verdicts account for the rest. A reasoning budget lifts a plain completeness judge from 0.570 to 0.670 paired. The same list plus a high budget still stops at 0.699, short of the pipeline's 0.786.
Restatement traces defeat every deployable method here. When the primary statement is gone and a full restatement survives elsewhere, the pipeline rule catches 0 of 57 and the evolved prompt 4 of 57. In 90.2% of those pairs the survivor sits under a different heading.
On real vendor notes, no benchmark threshold transfers. After re-calibration the evolved prompt still detects more than the best monolithic judge at about half the false-alarm rate. Of the pipeline flags on notes with a panel-verified omission, 75.6% name the very fact the census panel verified.
If an LLM is the quality layer on an ambient scribe, it is nearly blind to the error class the audits say dominates. Prompt wording, voting, and prompt search do not fix that. The task has to become: list the facts the transcript established, check each one against the note, and decide from per-fact verdicts rather than from an average coverage score.
The two operating points buy different things. The pipeline is expensive and quiet, and each flag is an instruction ("check whether the clindamycin allergy is in the note"). The evolved prompt is about a tenth of the price and catches more notes, but the flag is a number below 10, and the threshold has to be re-established on each deployment's clean notes. A coverage-scored pipeline will miss fabrications, so a faithfulness judge still has to run beside it.
This is an incremental repair. The best method still catches a little over a third of notes that contain an omission.
The omission arm has no external physician-labelled anchor. MEDEC anchors commissions; omission results remain internal to this instrument. Injected pairs are cleaner than production failures: on real vendor notes both detection and false alarms jump, so the published operating points do not transfer. Headline numbers come from one judge family; the asymmetry replicates on a second family, single-note detection does not. The clinician who graded disagreements is an author, and the structurally blinded stages hold 6 to 20 items each. The corpus is English UK and US primary and ambulatory care, with no EHR structured fields. The authors sell evaluation tooling; the pipeline they recommend is their own design. Prompts and judgements are released.