2026-08-04
623 laypeople and 153 doctors tested a dermatology AI: accuracy and skin-tone fairness rose, but LLM explanations misled laypeople when wrong (−21%) while doctors stayed resilient.
AI is moving into medicine, but the models are black boxes, so what lets a doctor or a patient trust a prediction? Explainable AI (XAI) is meant to fill that gap: GradCAM overlays a heatmap on the image regions the model fixated on, CBIR surfaces the most similar past cases, and a multimodal LLM just writes a rationale in plain text. The premise is that showing the reasoning lets humans decide whether to follow it.
Prior work already turned up a paradox: explanations can make people lean on AI more, not less. This paper asks a sharper question: when the explanation comes from today's multimodal LLMs, does it affect experts and laypeople the same way? It does not, and the gap is large.
Two large-scale online experiments. Study 1 recruited 623 laypeople for a melanoma-versus-nevus binary task (easier); Study 2 recruited 153 primary care physicians (PCPs) for open-ended differential diagnosis (harder), plus 320 medical students as a middle comparison. Each saw 12 clinical images, balanced across skin tone (Fitzpatrick 1–6) and pathology.
The model itself is a fairness-constrained dermatology network trained with CDANN (conditional domain-adversarial training) so its outputs stay balanced across skin tones. Accuracy was held at 83.3% in Study 1 (10 right, 2 wrong) and about 79.2% in Study 2.
Two factors were crossed. The first is explanation type: basic (diagnosis only), CBIR, GradCAM, or a multimodal LLM text rationale. The second is decision order: Human-First (decide, then see AI) or AI-First (see AI, then decide). Outcomes: accuracy, confidence-accuracy calibration, and skin-tone disparity.
The fairness model held up. In Study 1 lay accuracy rose from 69.7% to 75.8% and the skin-tone gap shrank by 76.9%; in Study 2 PCP top-1 rose from 11.5% to 33% (+21.5 points), top-3 by 43.5 points.
The split is in what happens when the AI is wrong:
| Group | AI correct | AI wrong |
| Laypeople (LLM) | +13.4% | −21.1% |
| PCPs (any method) | steady gain | barely moves (β=0–0.021) |
For laypeople the LLM explanation is the biggest booster when the AI is right, but also the biggest drag when it is wrong (−21.1%, versus −14.6% for basic). That is automation bias: people follow the model even when it errs. PCPs invert this. They gain when the AI is right and stay almost unmoved when it is wrong, leaning on their own judgment. Medical students fall in between: the more senior, the more error-resistant.
Order matters too. Putting AI first (AI-First) amplifies deference; in PCPs the LLM explanation raised the share of deferential participants by 19 points.
For anyone building medical AI or human-in-the-loop products, three practical lessons land. First, LLM explanations are not a universal good and can hurt non-experts: a confident paragraph reads to a layperson as the right answer. Second, do not surface the AI diagnosis before the human decides; anchoring magnifies deference, and Human-First is the safer order. Third, fairness-constrained training (CDANN) works, and its skin-tone gains survive into the final human-plus-AI decision, not just the model's own scorecard.
Same AI, different audience, different explanation, different order: that is what decides whether it helps or harms.
The authors flag several: limited ecological validity (no real clinical context such as age or history, simplified tasks, only 12 images each, online rather than bedside); LLM explanations generated in a single pass with no filtering or refinement, with a confident-tone prompt as a confounder; results specific to their model and XAI implementation.
A sharper concern: model accuracy was fixed at 83.3%, but real systems drift, and the strength of automation bias likely tracks how confident the model sounds, so the more confident it is the more it misleads, which the paper does not unpack. Twelve images is thin for individual-level deference analysis. And because the fairness gains come from model design rather than XAI, the net value of LLM explanations for laypeople is genuinely murky: they amplify the model's correctness and amplify its errors by the same lever.