ChemCrow: Augmenting large-language models with chemistry tools
Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, Philippe Schwaller
physics.chem-ph, stat.ML
2023-04-12
ChemCrow couples GPT-4 with 18 expert chemistry tools, autonomously planning and executing syntheses of an insect repellent and three organocatalysts and guiding the discovery of a novel chromophore, yet GPT-4 as evaluator cannot tell fluent wrong answers from correct ones.
Computational chemistry has accumulated excellent tools over decades: retrosynthesis planners, reaction predictors, property lookups. Each has its own interface and steep learning curve, so outsiders cannot use them, while LLMs, for all their generality, freely invent molecules and reaction conditions when asked chemistry questions. This EPFL work wires the two together: GPT-4 as dispatcher, expert tools as hands.
ChemCrow is a ReAct-style loop built with LangChain: the model writes a Thought, picks a tool, reads the Observation, and continues. The 18 tools fall into four groups:
Safety is a hard gate: every synthesis task passes the controlled-substance and explosive checks first, and a hit halts execution. Asked to synthesize nitroglycerin, the system refuses and issues a warning.
| Task | Outcome |
| DEET insect repellent | Autonomous planning and execution on IBM RoboRXN; mid-run errors such as insufficient solvent were fixed by an LLM-driven correction loop without human help |
| Three thiourea organocatalysts (Schreiner, Ricci, Takemoto) | All syntheses completed autonomously |
| Novel chromophore | Given a 369nm absorption target, trained a Random Forest to screen candidates; the suggested molecule measured 336nm and was confirmed as a new chromophore |
| 14-task evaluation | Four expert chemists preferred ChemCrow overall, with the edge growing on complex tasks; GPT-4 won only on easy memorization-heavy ones such as DEET and paracetamol synthesis |
The last row hides the paper's most interesting finding: GPT-4 as evaluator cannot distinguish clearly wrong GPT-4 completions from ChemCrow's work. In one concrete case, the Takemoto catalyst synthesis, GPT-4 proposed a multi-step route with unnecessary protection and deprotection steps risking the chiral center, while ChemCrow proposed a well-rated single-step route with conditions. The evaluator GPT-4 preferred the flawed route and praised its supposedly deep understanding.
For AI4Science practitioners this is the canonical template for tool-augmented LLMs: the model carries no chemistry knowledge, only dispatch and a natural-language interface, while tools supply the facts. The chromophore case shows the shortest path to human-AI discovery of new materials. The evaluation finding deserves equal attention: LLM judges favor fluency, and that bias cannot be ignored when the judged objects are LLM outputs.
The authors list four: tools do not fully fix faulty LLM logic; LLM-based evaluation is unreliable, leaving slow small-scale expert review; closed-source API models hurt reproducibility; performance is bounded by the underlying tools, so ChemCrow cannot beat what it calls. Task selection may carry implicit bias.
The reproducibility test is instructive: five independent runs of one task all produced correct products, but in two of the five the model misread a SMILES string as a trans-alkene, derailing the mechanism explanation. Execution is stable; explanation is not.