Base-model probes predict cross-benchmark coding-agent potential at ρ=0.976
hyunw_kim · x · 2026-10-20
NVIDIA researchers report that their DeepSWE-derived decisive-action probe tracks future SWE-bench Multilingual rankings at ρ=0.976, and prefix pass@32 tracks Terminal-Bench 2.1 at ρ=0.930. Traditional benchmarks fare far worse — HumanEval even shows ρ=-0.394. The signal transfers across benchmarks: best traditional coding performance today doesn't equal best future coding-agent potential.
More from Research
- DeepMind researcher warns AI-only investment could leave science with more hypotheses than labs to test them — weballergy · 2026-10-20
- New podcast: extending DeepLIFT attribution to AlphaGenome and beyond — anshulkundaje · 2026-10-20
- Spatial transcriptomics tools STalign/STcompare ported to C++, hours to seconds — anshulkundaje · 2026-10-20
- Dependently typed spreadsheet prototype surfaces in debate on proving Excel correctness with Lean — rickasaurus · 2026-10-20
- SOAP & SPlus scale pretraining across 96+ DiLoCo workers via update whitening — aaron_defazio · 2026-10-20
- World Models and Interpretability Are Two Sides of the Same Coin: Sonia Joseph Podcast — soniajoseph_ · 2026-10-20