2026-09-15
A fully local clinical agent matches GPT-5.2 within 0.7 points on MIRA-v2 (Qwen-3.5 at 90.0%); gating on five-run diagnostic consistency keeps 49.4% of cases at 98.9% accuracy.
Hospitals that want a clinical agent to finish a diagnostic workup without a human in every loop hit two separate trust problems. Operational trust is whether records, models, and audit logs can stay inside the institution. Decisional trust is whether this particular output should go through or go to a clinician. Cloud models already run multi-turn history-taking and tool use on MIMIC-derived benchmarks. Sending protected health information off-site usually fails governance. Token likelihoods and a few hedging phrases, measured on single-turn QA, also fail to track errors that amplify across tool calls.
The Kather group at TU Dresden tied both problems to one fully local stack: can open-weight models stay close to a cloud baseline, and can inference-time reliability scores pick a lower-risk subset for autonomous handling.
The system is a dual-agent simulator adapted from MIRA. A Physician Agent interviews, calls tools, and submits one diagnosis (Dx) plus a reasoning trace (R). A Patient Agent is grounded in a case summary and, in the baseline, is forbidden from inventing unsupported findings. Tools cover physical exam, labs, urinalysis, radiology reports, microbiology, prescribing, and procedure requests. Encounters cap at 10 turns and end on an admission action. The original MIRA Plan tool was dropped to save context for on-premise models.
GLM-4.5-Air at temperature 0.01 is the reliability-analysis workhorse. Qwen-3.5, GLM-5, and GPT-OSS were added under the same agent code; GPT-5.2 is the cloud control. Serving is vLLM on NVIDIA H200 and RTX PRO 6000 GPUs. MIRA-v2 correctness is judged by a separate Gemini 3.1 Flash-Lite; CDM keeps the published fuzzy matcher.
Each case is sampled N=5 times. Four scores are computed on both Dx and R:
Selective autonomy is a threshold: keep the case if the score is high enough, else defer. A stress test strips the Patient Agent of chief complaint and history so it has to confabulate.
On MIRA-v2 (seven diseases, n=551), Qwen-3.5 reached 90.0% accuracy, GLM-5 89.7%, GLM-4.5-Air 88.4%, GPT-OSS 85.3%. GPT-5.2 sat at 90.7%. The best local model is 0.7 points off the cloud line. Pneumonia was the weakest class for every model.
On CDM (four abdominal diseases, n=2,400), Qwen-3.5 hit 83.8% and GLM-4.5-Air 81.2%, against a prior open-weight leaderboard top of 70.5% (Gemma-3). Residual errors clustered in clinically overlapping pairs: pancreatitis vs cholecystitis, appendicitis vs diverticulitis.
Blinded physicians reviewed 181 cases. 81.8% were judged clinically valid for both the agent and the EHR label; 4.4% were valid for the agent only. Agreement with the LLM judge was 92.3% (Gwet AC1=0.898). A MedGemma differential-diagnosis critic moved overall accuracy from 88.4% to 87.1%, so later analyses stayed single-agent.
| Signal | Stream | AUC |
| ConsistencyDx | diagnosis | 0.860 |
| ProbScoreDx | diagnosis | 0.747 |
| LingCertR | reasoning | 0.792 |
| ConsistencyR | reasoning | 0.785 |
| ConceptDensityR | reasoning | 0.426 |
DeLong's test on ConsistencyDx vs ProbScoreDx: difference 0.114, P=0.0011. Along the high-consistency row, accuracy sat at 99.0–100.0% regardless of internal probability. ConceptDensityR ran below chance and should not be read as confidence.
Ungrounding the patient dropped accuracy from 90.6% (499/551) to 70.2% (387/551). ConsistencyDx fell with accuracy and still discriminated (AUC=0.875). For pneumonia, ProbScoreDx and LingCertR rose above 0.90. Internal likelihood did not track lost evidence.
At ConsistencyDx ≥ 0.90 the agent kept 272/551 cases (49.4%) at 98.9% accuracy, with three residual autonomous errors and 49 errors packed into review. ProbScoreDx ≥ 0.98 kept 22.5% of cases at 97.6% and never reached the same accuracy band. Raising N from 3 to 10 moved retained accuracy from 97.5% to 98.9% to 99.6%, and coverage from 51.7% to 49.4% to 46.3%. Five runs cost about five times the tokens.
VivaBench (n=990, ten specialties, PubMed case reports) is harder: Qwen-3.5 72.22±0.92%, GPT-OSS 67.96±1.09%. ConsistencyDx remained the strongest signal (AUC=0.719). A 0.85 threshold yielded 89.9% accuracy at 32.0% coverage.
The usable package for a hospital is local open-weight models that nearly match GPT-5.2 on a MIMIC emergency benchmark, plus a consistency gate that lets about half the cases through while concentrating errors in review. Self-consistency here is a routing signal, not a decoding trick. Thresholds and model upgrades stay under institutional control.
The gate is expensive. Consistency needs several full multi-turn encounters. Absolute thresholds move with temperature, encoder, and N: at 0.85, BGE retained 73.9% of cases versus 62.6% for MiniLM, without a retained-accuracy gain. Do not copy a number across configs.
Both primary benchmarks come from one MIMIC-IV site. The study is text-only; native image reading is out of scope. The critic agent did not help overall. High-consistency residuals remain: some are timepoint mismatches between the presenting syndrome and a later etiologic label, some are premature closure on a familiar narrative. Older patients had lower accuracy; a fixed threshold may send a higher-risk group down a different autonomous/review mix, and the paper flags that as descriptive. Everything is a retrospective simulation. There is no prospective workflow and no measurement of real clinician reliance. Serial latency scales like N.