Hiding Random Data Points Is a Weak Test of Reaction Mechanisms, Chemists Warn
bravo_abad · x · 2026-11-18
- In chemistry ML, "hide a data point and predict it back" is a common test for whether a model has learned a reaction mechanism — but it's a weak one.
- The catch is which points you hide: reaction curves are smooth, so a hidden point sits between two visible neighbors, and even a simple curve fit with no chemistry in it predicts it correctly. Passing says nothing about the mechanism.
- Jakob Dahl and Klavs Jensen demonstrate this in a Digital Discovery perspective using their earlier palladium-catalyzed reaction data: two rival mechanisms with the same number of adjustable parameters can both fit well under random held-out tests.
- The takeaway: held-out tests must be designed so competing mechanisms make distinguishable predictions, not chosen at random.
More from Research
- AIR-O decodes visual attention from brain scans in real time, generating images noninvasively — jonxuxu · 2026-11-18
- Goodfire's Neuron Communities: a handful of neurons in LLMs can handle arithmetic and formatting — fatihdin4en · 2026-11-18
- JAMA study finds AI-generated discharge summaries match clinician-authored ones on harm potential — EricTopol · 2026-11-18
- LoRA dial can precisely fake low benchmark scores: up to 97% of faked errors land on one letter — rohanpaul_ai · 2026-11-18
- Study finds latest Claude models were trained on protein sequences at scale — LeoTZ03 · 2026-11-18
- Reviewer fatigue in the coding-agent era: "everything is possible, nothing is interesting" — rishabh16_ · 2026-11-18