ASI-Bench evaluates AI's autonomous scientific research capabilities
rohanpaul_ai · x · 2026-08-25
The post introduces the paper 'ASI-Bench: At the Dawn of Artificial Superintelligence.' Built by over 40 experts across 31,000+ hours, this benchmark features 60 real-world research projects across 11 scientific domains. It progressively withdraws human methodological guidance to test how far AI can proceed on its own—comparing performance when given full procedural steps versus just a method name. The author notes that specifying the procedural steps carries significantly more performance weight than merely naming the method.
Related event: Tsinghua and Partners Release ASI-Bench to Test AI Scientific Autonomy(3 posts)→
More from Research
- False Collatz Proof via a Lean Bug: Talk Probes Proof Assistants as LLM-Found Bug Targets — marc_lelarge · 2026-10-16
- Goodfire's Nick Cammarata argues neural networks are 'already clean' in new interpretability essay — ctjlewis · 2026-10-16
- NEJM AI: prompt tuning cuts LLM clinical test-order cost ratio to 1.4x across 19 systems — dvir_a · 2026-10-16
- Prompting cuts LLM over-testing costs across 19 systems, but diagnosis agreement drops — dvir_a · 2026-10-16
- AI Recommends 3.2x Costlier Care; Models Add Tests in 2/3 of Visits Physicians Ordered None — dvir_a · 2026-10-16
- 24 AI Systems Recommend $314 Per Visit in Care vs Physicians' $97, a 3.2x Gap — dvir_a · 2026-10-16