What's in a Name? Morphological Shortcuts by LLMs in Pharmacology
Kaijie Mo, Thomas Yang, Chantal Shaib, Qing Yao, William Rudman, Ramez Kouzy, Kanishka Misra, Byron C. Wallace, Junyi Jessy Li
cs.CL
2026-06-04
Nine LLMs treat fake affix-bearing names as real drugs. Of 653 drugs, Qwen marks 206 as affix-driven; medical fine-tunes overgeneralize most. The shortcut sits in layers 2-10.
The WHO INN system encodes drug class in the name: -cillin marks penicillins, -ciclib marks CDK inhibitors. A human seeing an unfamiliar name will guess the class and hedge. An LLM will write that guess as clinical prose, at the same confidence as a memorized fact.
That is shortcut learning in a high-stakes lexicon. Form and class co-occur so consistently in training that affix statistics can stand in for drug-specific knowledge. Prior work already showed LLMs treating Pokémon names as medications and inventing clinical details. Nobody had measured whether the official naming system itself can drive the hallucination.
The stimuli come from the American Medical Association list of 655 medical affixes. Two irregular real-drug examples drop out, leaving 653 matched triplets:
Nonce stems are absent from Wikipedia and Merriam-Webster, then checked on Google so they are not attested drugs.
Two behavioral tasks. Multiple choice asks "What is {drugname}?" with four shuffled options: the affix-consistent class definition, two unrelated definitions, and "non-existent drug." Open-ended prompts look like user questions, e.g. "Is {drugname} effective for treating {condition}?", judged by Claude-Sonnet-4.5 on whether the model treats the name as a real medication. On 300 double-annotated cases the judge hits 92.3 accuracy and 89.3 F1. Nine models are tested: Gemini-3-Flash, Gemini-2.5-Flash, GPT-5-mini, Llama-3.1-70B-Instruct, OLMo-3-7B-Instruct, OLMo-3-7B-Think, Qwen2.5-7B-Instruct, plus two medically fine-tuned Qwen variants, HuatuoGPT-o1-7B and Meditron3-Qwen2.5-7B.
For real drugs they run a 2x2 perturbation. RR is the original name; NR swaps the stem; RN swaps the affix; NN swaps both. Probabilities over those four conditions yield AffixScore, StemScore, and HolisticScore. Drugs then fall into holistic, affix-dependent, stem-dependent, mixed, or no-signal. No-signal requires the RR-vs-NN gap to be unstable (mean-std ≤ 0.1). A dominant label needs a 0.1 gap over the runner-up.
Mechanistic work uses activation patching on OLMo-3-7B-Instruct and Qwen2.5-7B-Instruct: restore a real affix or stem activation into a fully nonce input and watch the target-token probability. 168 cases, 56 affixes. An appendix trains a rank-1 DAS direction per layer to steer "classify by affix" versus "call it non-existent."
Fake names with real affixes push models toward class-level pharmacology. On multiple choice the effect grows with scale (Llama-3.1-70B, Gemini). Open-ended answers polarize: medically fine-tuned models, especially Meditron3-7B and HuatuoGPT, overgeneralize the hardest, often treating Real, Fake, and even Nonce strings as valid drugs. Nonce stays more conservative than Fake, so the real affix is doing the work. Chain-of-thought makes models more cautious: fewer fake-as-real errors, and a small drop in recognizing real drugs. The stickiest affixes are inhibitor classes (PARP, EGFR, KRAS, MEK). Sticking a real affix on an everyday word (tablecillin) still works. OLMo-3-7B-Think is the most conservative of the set.
The 653-drug diagnostic is sharper. On Qwen's multiple-choice run, 206 drugs are affix-dependent, 157 holistic, 250 no-signal. OLMo has 384 no-signal and 120 affix-dependent. Holistic drugs behave most reliably: OLMo generation accuracy 76.3 on MC, versus 20.6 for no-signal. Open-ended tasks shift mass from affix-dependent to holistic: class ID can ride on morphology, drug-condition reasoning wants the whole name.
| Model | Task | Holistic n / Gen | Affix n / Gen | No signal n / Gen |
| OLMo | MC | 115 / 76.3 | 120 / 74.2 | 384 / 20.6 |
| Qwen | MC | 157 / 78.8 | 206 / 80.1 | 250 / 13.2 |
| Meditron | MC | 152 / 88.7 | 204 / 84.8 | 242 / 24.4 |
In the OLMo-3-7B-Instruct training corpus, HolisticScore correlates with real-drug frequency (Spearman ρ=0.17 on MC, 0.19 on OE). AffixScore on MC goes the other way (ρ=-0.21). Rare drugs get guessed from form.
The models almost never say they are using the affix. On 70 affix-dependent real drugs, OLMo picks the affix-consistent definition 53 times without mentioning the affix, and mentions it 3 times; Meditron is 64 vs 1. Under NR, where the affix is the only cue, Meditron still picks the definition 57.1% of the time and Qwen 47.1%, with little explicit morphology talk. Case studies: antazoline (AffixScore 2.27) borrows mechanisms from naphazoline; seliciclib (1.43) imports targets and trial evidence from other -ciclib drugs. Holistic-dominant minocycline and zolpidem do not show that transfer.
Patching pins the shortcut to early-mid layers (about 2-10) at the last subject token, then to the final token later. For affix-class drugs, NR→NN and RR→NN curves nearly overlap: the affix carries most of the causal effect of the full name. DAS peaks at layers 7-10. A single rank-1 direction captures about 85% of the KL, flips 45% of top-1 predictions. On 150 held-out NR examples, layer 8 at |α|=5 flips 84% of strong definition-preferring cases to "non-existent" and 76% of strong non-existent cases to the affix definition. A random direction and layer 23 do almost nothing; a mean contrastive vector flips only 30%.
If you ship a medical LLM, pharmacy RAG, or dosing chatbot, naming the class is not knowing the drug. Affixes are a legitimate training signal; WHO designed them that way, so the shortcut looks like competence. Medical fine-tuning does not remove it. Meditron3-7B still scores 83.3 generation accuracy on no-signal strings in the open-ended task, treating almost anything as a drug.
The 2x2 audit is reusable. The early-mid causal direction is a concrete intervention target. Neither is a production patch yet, and closed models cannot even be diagnosed this way.
The authors flag two: the diagnostic needs logits, so most closed models are out; mechanistic experiments ran on two 7B models only. Gemini and GPT-5-mini appear in the behavioral plots, not in the diagnostic or the patching.
The open-ended "treats as real" label is an LLM judge. F1 89.3 on 300 items leaves about 10% noise in the overgeneralization rates. The AMA list gives one example drug per affix, so 653 covers the affix table, not the long tail of brand names in real prescriptions. CoT cuts fake-as-real errors and also cuts real-drug recognition. Retrieval or tool use is not tested as a mitigator.