ChefNMR2 benchmark: 82.3% top-1 accuracy on 2,000+ molecules vs 52% for best LLM
ZhongingAlong · x · 2026-11-03
The author's team teamed up with Edison Scientific to benchmark the newly released ChefNMR2 against specialized structure elucidation tools and 6 frontier reasoning LLMs across 5 test sets and 2,000+ molecules, including experimental NMR data.
- ChefNMR2 hits 82.3% top-1 structural accuracy; the best specialized de novo model manages 35.4%, the best retrieval method 60.1%, and the best LLM 52.0%.
- The chart shows ChefNMR2 leading on all five datasets (SpectraNP+, USPTO, NMRexp, SpecTeach, NMRShiftDB2); compared LLMs include Claude Opus 5, GPT-5.6-Sol, Gemini-3.6-Flash, Kimi-K3, GLM-5.2, and DeepSeek-V4-Flash.
- Notably, several LLMs score suspiciously high (96-100) on datasets likely seen in training, hinting at benchmark contamination; the gap is widest on truly unseen experimental data (NMRexp).
- The author argues that since Claude Opus 4.8 in June, LLMs have suddenly become good at interpreting NMR spectra like a chemist — a sign LLMs are now capable of real scientific tasks.
Related event: ChefNMR2 Benchmarks: 82.3% Top-1 Accuracy vs 52% for Best LLM(3 posts)→
More from Models
- Spider-Verse-style 2D animation built with Opus fuels rumors it routes to Fable 5.5 — minchoi · 2026-11-03
- More "Fable 5.5" demos: pelican, horse and bike scenes drawn and animated entirely in code — minchoi · 2026-11-03
- One prompt, four models: rumored "Fable 5.5" ships a full 3D open-world RPG while rivals stumble — minchoi · 2026-11-03
- Rumored Claude Fable 5.5 beats Opus 5.5 and Gemini 4 Argon on three.js giraffe test — minchoi · 2026-11-03
- Fable 5.5 zero-shot generates interactive 3D sailing ship; launch rumored Tuesday — minchoi · 2026-11-03
- Qwen base model looped on 93% of MATH-500, casting doubt on claimed 6x gain — paul_cal · 2026-11-03