ChemCost Benchmark: Best Agents Hit Only 50.6% Estimating Chemical Purchase Costs
jiqizhixin · x · 2026-11-13
ChemCost, from Carnegie Mellon, Notre Dame, UNC Chapel Hill and Pittsburgh researchers, turns chemical procurement cost estimation into a reproducible agent benchmark. A frozen snapshot covers 1,427 evaluable reactions, 2,261 chemicals, and 230,775 supplier pack quotes. Agents must resolve identities, retrieve quotes, pick compliant packs, and normalize cost per product mass at one-millimole limiting-reagent scale (solvents excluded). The best reported clean-input result reaches only 50.6% accuracy within 25% relative error, with controlled perturbations (ambiguous names, quantity chaos) making it harder still — far beyond mere chemical-name recognition.
More from Research
- Carl Edwards' AssayBench talk: predicting biology and what to measure next — kchonyc · 2026-11-13
- Agent-Generated Heatmap Shows Inkling 2 Small's Split Q/K/V Queries Diverge in Later Layers — stochasticchasm · 2026-11-13
- Study of 135 model releases: AI labs barely share safety benchmarks, top one used by only half — ChenhaoTan · 2026-11-13
- Stanford Paper Shows Full Disclosure Can Hurt LLM Trading Agents Across 10 Frontier Models — shangbinfeng · 2026-11-13
- UMI diffusion policy hits 100/100 cube-stacking success with slowed demos — DominiqueCAPaul · 2026-11-13
- Wharton's Generative AI Lab open-sources Primer, an agent system turning teacher talk into classroom simulations — emollick · 2026-11-13