2026-08-24
MedGuard (Qwen2.5-7B) checks Chinese telemedicine claims, retrieving only when unsure. Fine-grained F1 is 0.7668 vs 0.6327 for Qwen3-235B; 126 clinicians scored all axes above 4.2.
Chinese telemedicine is mostly text, and a single consult can stretch over hours or days. Diagnoses and prescriptions sit in fragmented threads, so a wrong premise is easy to miss. Most medical LLM work still scores static QA or simulated visits, with safety reduced to a prompt. Used as a live alert, those systems hit three walls: long transcripts bury the sentences that need checking; sycophancy floods the clinic with false positives; hallucinations invent plausible pharmacology that a doctor cannot trace.
Xiamen University and Ant Group treat this as a gatekeeper problem. The system flags claims in diagnoses, prescriptions, and dialogue that look incomplete, ambiguous, wrong, or misleading. It does not try to replace the clinician.
The backbone is Qwen2.5-7B-Instruct. The full agent is trained end-to-end with GRPO. External knowledge comes from six Chinese sources (Dingxiang, Tencent Medical Encyclopedia, Xiaohe, a clinical decision assistant, Baidu Health, and the Chinese Pharmacopeia): 16,535 disease entries and 187,738 drug entries, indexed with BM25.
The unit of work is a claim, not a transcript.
Labeled agent trajectories are scarce. Two thousand Huatuo encyclopedia QA pairs are rewritten by DeepSeek-R1-671B with contextual perturbation and label inversion, yielding a balanced risk / no-risk set. Reward has two channels: JSON format must be valid, otherwise a −1 format penalty; if valid, the outcome is +1 for a correct risk, 0 for a correct no-risk, −1 for a miss or a false alarm. Training used up to eight A100 80GB GPUs. The paper reports about 4 GPU-hours for a full run.
MedGuardEval has 1,665 consults and 7,589 human atomic claims, all out of the training domain. REAL is 235 HaoDF threads. CASE is 682 dialogues rewritten from public clinical cases. DRUG is 748 dialogues built from NMPA labels. The headline metric is strict: every claim must be extracted and every risk label must match.
Baselines are six open models and three agent stacks (CoM, OctoTools, HealthCareAgent) on the same 7B backbone. Those agents sit at overall fine-grained F1 0.44–0.60. Against the strongest large-model baseline, Qwen3-235B-A22B:
| Metric | MedGuard | Qwen3-235B-A22B | Gap |
| Precision | 0.6827 | 0.5535 | +0.1292 |
| Recall | 0.8746 | 0.7384 | +0.1362 |
| F1 | 0.7668 | 0.6327 | +0.1341 |
Average lifts versus larger models are reported as 24.8% precision, 17.6% recall, 22.1% F1. Recall across the three subsets stays in 86.1%–91.2%. Stratified by ATC class, semantics, age, and body system, mean F1 is 0.77 against 0.62 for Qwen2.5-72B and DeepSeek-R1-671B. MedGuard is top-3 in 38 of 41 slices and first in 22. Versus SOTA agent systems the abstract cites a 21.20%–23.02% average edge.
On real consults, antiparasitic and sensory-organ drugs show risk rates 83.33% and 80.00%, against 42.31% and 35.70% for blood and miscellaneous classes. Neonates 62.50% and infants 60.94% sit above adults at 58.96%. Cases with complications 61.05% versus 34.79% without; cancer pathways drop to 16.69%.
One hundred twenty-six licensed clinicians from ten departments scored 604 HaoDF consults. All seven Likert axes have means above 4.2, with 3 predefined as usable. Published six-axis means: factual accuracy 4.389, time compression 4.402, information gain 4.295, communication readiness 4.377, error reduction 4.284, legal-ethical safety 4.332. The share scoring above 3 is 83.60%–92.80%. Same-score ratio 0.62–0.67; coefficient of variation 0.22–0.24.
A faithfulness audit on 300 claims perturbs facts. Inserting a risk into a no-risk claim raises the Risk Conviction Score by 1.03 on average, up in 81.8% of cases. Repairing a risk claim lowers it by 0.94, down in 84.0%. On 10,000 unlabeled HaoDF records MedGuard fires 2,234 alerts against 4,785 from DeepSeek-R1-671B, with mean entity density 0.0105 versus 0.0072. Airway emergencies sit +13.14% above the mean identification rate; polypharmacy +0.1132. That slice has no gold labels, so the comparison is volume and density, not F1.
A 7B model with retrieval and a cautious split beats 235B/671B direct answers and generic agents on this Chinese claim-checking task. The reusable pieces are atomic decontextualized claims, retrieve-only-when-unsure, and GRPO on format plus outcome. The knowledge base and the three annotated subsets are useful even if the agent is swapped. Clinician scores say the explanations are readable and that false positives did not push the tool below the usability line.
The target is verifiable claims, not bedside manner, empathy, or ethics. On the 10k set DeepSeek fires more than twice as many alerts. MedGuard treats higher entity density as a proxy for precision. Without labels that proxy stays a proxy.
The authors flag the obvious: text only, no images or tables; a support tool, not a decision-maker; prospective deployment still needed. The evaluation design deserves more weight. Candidate cases that DeepSeek-R1 or Qwen3-235B already solved were dropped, so the remaining set is enriched for failures of those models and the gap against them is inflated. CASE and DRUG are LLM-written. Claim matching also uses DeepSeek-R1, so generator and judge overlap. REAL has 235 threads. The doctor study reports absolute Likert scores for MedGuard, with no paired scores for a competing system in Table 8. Four GPU-hours of GRPO is unusually short; there is no learning curve and no multi-seed report. The knowledge base includes commercial encyclopedias whose entry quality was not independently audited.