Tsinghua's SRHarness and MechBench reveal AI agents find formulas but miss the mechanisms behind them
机器之心 · wechat · 2026-10-14
Tsinghua University's Data Science and Intelligence Lab released two AI-for-Science works: SRHarness, a runtime framework for agentic symbolic regression, and MechBench, a benchmark for mechanistic discovery — asking whether AI can reliably find formulas and whether it truly understands the mechanisms behind them.
- SRHarness unifies reusable scientific actions, persistent scientific state, and trajectory lifecycle management. It hits 93.69% symbolic recovery on LSR-Transform (+30 points over prior SOTA SR-Scientist) and 72.97% on anonymized tests with a DeepSeek-v4-flash base — far above Codex's 20.72% — matching Codex+GPT-6-Astra with a lighter model. Open-sourced on GitHub/PyPI with an online demo.
- MechBench explicitly separates recovering phenomenal laws from recovering underlying mechanisms, built from 16 real mechanism task families across physics, chemistry, biology, and materials (512 questions), with mechanism probes and competitive-mechanism screening to ensure objective, non-memorizable evaluation.
- Key finding — a capability gap: Codex (GPT-5.6-sol) recovers 35.00% of phenomenal laws on the core set but only 13.75% of mechanisms; even when handed the correct law, mechanism recovery stays under 50%. GPT-6-Astra lifts law recovery to 72.50% yet manages only 47.50% mechanism recovery.
- The work provides a quantifiable yardstick for whether AI scientific discovery can move from "fitting observations" to "understanding mechanisms," arguing future scientific agents will require co-design of models, search strategies, and runtime frameworks.
More from Research
- Stanford and Databricks use AI agents to reproduce medical findings and test analytical robustness — ChengleiSi · 2026-10-23
- EMNLP paper: diffusion LMs internally encode latent denoising timestep representations — gsarti_ · 2026-10-23
- KU Leuven and VIB open two Group Leader professorships in AI for Biology, start 2027 — jpillowtime · 2026-10-23
- Subspace Tuning curbs LLM catastrophic forgetting by freezing principal singular directions — scottniekum · 2026-10-23
- Perceptic launches PercepticOS, an AI workspace for 'scientific superintelligence' — nathanbenaich · 2026-10-23
- EPFL's Bosselut to talk long context as test-time learning at ELLIS seminar — andre_t_martins · 2026-10-23