ResidencyRL: Reinforcement Learning in Simulated Clinical Environments
Valentin Liévin, Samuel Schmidgall, Tim Strother, Alex Bijamov, Akshay Goel, Anil Palepu, Chunjong Park, Vahid Balazadeh, Min Woo Sun, Marius Guerard, Justin Chen, Dave Steiner, Vikram Dhillon, Ibrahim Azar, Akhil Mehta, Nicholas Spetsieris, Shilpan Shah, Maen Abdelrahim, Amit Dahiya, Yun Liu, Katherine Chou, Yossi Matias, Avinatan Hassidim, Dale R. Webster, Quoc V. Le, Raia Hadsell, Joelle Barral, Carey Radebaugh, Aleksandra Faust, Shekoofeh Azizi, Mike Schaekermann, Po-Hsuan Cameron Chen, Tao Tu, David Racz, Lin Yang
cs.AI, cs.CL
2026-08-08
Trains a Gemini clinical agent with GRPO in simulated multi-turn encounters: adversarial diagnostic accuracy 81% to 88%, missed red flags down 31%, 87.6% expert preference.
Real clinical reasoning is not a multiple-choice exam. A clinician has to extract a history across many dialogue turns, revise diagnostic hypotheses on the fly, and decide management under uncertainty. LLMs already score well on static medical benchmarks such as MedQA, but almost nothing optimizes the full sequence of decisions inside a patient encounter. The most common diagnostic error in practice is "premature closure": locking onto a diagnosis after two or three questions and stopping the workup early. ResidencyRL, from Google DeepMind and Google Research, makes that full sequential encounter trainable.
The core is a simulated clinic the agent can rehearse in, plus a reinforcement-learning loop.
The simulated environment has several parts. A scenario pipeline samples demographics from US distributions, uses Gemini 3.1 Pro to generate clinical cases (chief complaint, history, ground-truth diagnosis), then quality-checks and deduplicates them with TF-IDF. A Gemini 3.5 Flash patient simulator answers from the case, with health-literacy adaptation and information asymmetry, so the patient does not volunteer the key detail unprompted. Adversarial "extension packs" add safety challenges. The policy (also initialized from Gemini 3.5 Flash) takes up to 60 dialogue turns plus 8 tool calls per trajectory; the tools are seven documentation instruments: primary diagnosis, differential, urgency grading, management plan, patient-facing summary, SOAP note.
The reward is the hard part. A primary reward Rprimary in [0,3] scores six dimensions over 26 Likert sub-axes: diagnosis (weight 2/9), management (3/9, covering urgency, investigations, treatment, follow-up, quality, safety), communication (1/9, via the PCCBP scale), documentation (1/9, via PDQI-9), intake history (1/9), and style (1/9, for example whether it asks one question at a time). A penalty Rpenalty covers red lines: hallucination, contraindicated actions, missed red flags, plus parsing errors and excess turns. Optimization uses Group Relative Policy Optimization (GRPO), with K parallel episodes per scenario for advantage estimation. The training mix is 49,870 telehealth scenarios, 5,000 history-taking, and 2,583 adversarial.
Two design choices stand out. The reward rubrics mirror real clinical-education instruments (PDQI-9, PCCBP) rather than invented proxies, and the adversarial patient is built to punish the agent for stopping the workup early.
The adversarial set shows the effect most clearly (n=200):
| Metric | Base | ResidencyRL |
| Diagnostic accuracy (rubric ≥4/5) | 81.0% | 88.0% |
| Missed red-flag rate | 45.5% | 31.5% |
| Missed critical-question rate | 65.5% | 43.5% |
On standard telehealth, management quality rises 3.98 to 4.52 (1-5 scale), urgency 93.5% to 98.5%, and social-history gathering 1.31 to 2.64.
The gains transfer. On the AMIE Mx multi-visit benchmark (120 scenarios, 360 visits), management reasoning (MXEKF) goes 80.1% to 88.4%, patient communication (GMPCQ) 83.7% to 92.2%, and clinical skills (PACES) 79.6% to 84.8%. External benchmarks move the same way: AgentClinic-MedQA (+4.2 pp), AgentClinic-MIMIC-IV (+6.5), and CRAFT-MD (+3.2 to +4.7).
In a blinded side-by-side read by 97 clinicians, experts preferred the trained agent 87.6% of the time (completeness of information gathering 90.7%, management-plan appropriateness 75.3%).
This is a reproducible recipe for turning a medical LLM that passes multiple-choice tests into an agent that can run an entire clinical encounter. The contribution is not one SOTA number; it is the reward design aligned to real clinical rubrics plus the adversarial simulator. It shows that the full sequence of clinical decisions can be learned with reinforcement learning, and that the learned skill transfers. For medical-AI teams, the reward-and-simulator blueprint is directly borrowable; for the broader agent community, it is evidence that long-horizon, multi-turn, tool-using agents can be trained with structured rewards.
The cold water: these are simulation results only. The authors repeat that prospective validation in real workflows is still required before any clinical use.
The authors note that the simulator does not model the downstream physiology of a prescribed treatment, that real-world prospective validation is missing, and that on adversarial cases 31.5% of red flags and 43.5% of critical questions are still missed, which is far from deployable. Two concerns stand out on a close read. The reward is auto-graded by an LLM judge over 26 sub-axes, and the paper does not deeply analyze how auto-grader bias might be amplified by reinforcement learning. Both the training and evaluation scenarios and patients come from the Gemini family, so there is a distributional closed-loop risk that independent real-world data would need to test.