2026-09-03
NIH NLM's Nature Protocols tutorial maps medical LLM use into five stages. A compiled MedQA table lists GPT-5 at 95.8%, with trial-matching notebooks.
Most medical teams still use LLMs by typing into a chatbot. This Nature Protocols tutorial, led by NIH's National Library of Medicine with several US universities, treats that habit as both underuse and misuse: ad-hoc prompting is not enough for medical work, and the failure mode can reach patients.
The missing piece is an operations manual, not another capability survey. The text walks through how to turn a clinical need into a model-shaped task, how to pick a model, how to change the prompt, when fine-tuning is actually warranted, and which compliance and fairness checks belong at deployment.
The workflow has five stages, each tied to a concrete best practice, and the running example is matching patients to clinical trials.
Tasks are typed into five buckets: knowledge and reasoning, summarization, translation, structurization, and multi-modal analysis. Collect about 100 labeled instances before development. Reasoning tasks can score short answers first (yes/no) and only then audit explanations. Summarization and translation may use BLEU, ROUGE, and BERTScore against references, with the explicit caveat that those automatic metrics often disagree with human judgment.
Model choice has three axes. Modality and context window are hard constraints. Medical skill is screened with exam-style benchmarks and confirmed with clinical evaluation. Access is a three-way trade among a web chat, an API, and a local deploy. Sensitive data goes through HIPAA: public OpenAI APIs are typically non-compliant; Azure and Anthropic offer compliant hosting; local Llama or Mistral keeps more control. Context is measured in PubMed abstracts: one abstract is about 300-400 tokens, Llama 3.0's 8k window holds about 20, GPT-5's 400k about 1,600, Claude 4.5 Opus's 200k about 800, Gemini 3.0's 1M about 2,500. Long windows still lose the middle of the prompt.
Prompting defaults to temperature 0 and JSON output. Use 1-5 diverse few-shot examples, starting from zero-shot. Ask the model to reason step by step before the answer. If knowledge is stale or invented, add RAG or tool calls. Prompts can also be optimized programmatically with DSPy or TextGrad. Fine-tune only when prompting stalls, abundant labeled data already exists, or the working prompt is too long to afford. PEFT / LoRA fits small, specific datasets; one clinical-summarization case used a few thousand examples on a single NVIDIA Quadro RTX 8000.
At deployment, outputs are support tools, not a substitute for a clinician. Keep watching fairness, bias, cost, and re-validation after a production model is retired.
This is a protocol paper. It does not run a new head-to-head experiment. The main artifacts are a compiled MedQA-USMLE (4-option) table and companion trial-matching notebooks.
| Model | Weights | Context | MedQA |
| GPT-5 | Closed | 400k | 95.8% |
| o1-preview | Closed | 128k | 94.9% |
| Gemini 3.0 Pro | Closed | 1M | 94.6% |
| DeepSeek-R1 | Open, 671B | 128k | 92.0% |
| Claude 4.5 | Closed | 200k | 91.4% |
| Llama 3.1 | Open | 128k | 88.2% |
| Llama 2 | Open | 4k | 47.8% |
Those scores are taken from papers and third-party boards, not re-run by the authors. Higher multiple-choice scores do not imply clinical utility, because real work has no options; exam screening still has to be followed by open-ended tests and, where possible, randomized trials. A cost sketch uses Gemini 3.0 pricing as of December 2025: $2 per million input tokens for prompts under 200k, $12 per million completion tokens. One thousand MIMIC-III discharge notes are about four million tokens, roughly $8 on the prompt side. 7-8B models often fit one A100 40GB; 70B usually needs multiple GPUs.
For clinicians and researchers who do not already own an eval stack, the five-stage map is usable. Typing the task first blocks "throw everything at the chat box." Putting HIPAA and local hosting inside model selection is earlier than bolting compliance on later. The companion notebooks turn tokenization, temperature, CoT, few-shot, RAG, and fine-tune data prep into runnable steps.
It is still an entry-level guide. On simple structurization, the tutorial itself says a fine-tuned BERT often matches or beats an LLM. Teams with a mature eval pipeline will mainly get a complete checklist and an updated 2025-2026 model roster.
The Nature article is subscription-only. This reading uses the authors' public arXiv manuscript and code companion; the abstract, four figure captions, and MedQA table match the journal page, but layout and some paragraphs may differ from the paywalled PDF.
The MedQA table is a collage, from system cards to third-party leaderboards, so cross-row comparisons mix setups. "Always use CoT" and "temperature 0" are rules of thumb, with no ablation on this tutorial's own medical set.
"About 100 test cases" follows prior eval papers and is thin for high-stakes clinical decisions. Fairness checks point at external bias benchmarks rather than measured gaps inside this workflow. The model list will age; lifecycle guidance only notes that prompts and validation have to be redone.