NVIDIA paper predicts post-training coding-agent potential from base models at ρ=0.964
hyunw_kim · x · 2026-10-20
NVIDIA researchers present "Before They Can Solve" (arXiv:2610.10478), a method to identify which base checkpoint deserves expensive agentic post-training — without running every base model as a full agent.
Problem: Base models often can't even operate a coding-agent harness. On SWE-bench Verified, 5 of 6 base models scored 0%, yet their post-trained versions reached up to 80.6%.
Method: Replay successful post-trained agent trajectories and rerun tests after each step to find the "decisive step" — the first action whose cumulative patch flips tests from fail to pass. Give the base model the context right before that step and measure with three probes:
- Decisive-Action BPB: likelihood of the correct action
- Patch MCQ: distinguishing passing from failing patches
- Prefix pass@K: generating a passing fix from the prefix
Results: Across 10 base checkpoints, Decisive-Action BPB tracks post-trained SWE-bench Verified rankings at ρ=0.964, and transfers across benchmarks (Multilingual ρ=0.976, Terminal-Bench 2.1 ρ=0.930). Traditional benchmarks predict far worse — HumanEval even ρ=-0.394. A model's future coding-agent potential is visible before post-training, if you know where to look.
More from coding & agent
- Dev turns his Goodreads shelf into a slick site you can rebuild with Claude — thisiskp_ · 2026-10-20
- Notion AI ships major upgrade: smarter Agents, Auto 90% cheaper, BYOK support — ivanhzhao · 2026-10-20
- iPhone Action Button + Grok voice + HeyGen MCP turns a sentence into an avatar video — HeyGen · 2026-10-20
- Vibe-coded game League of Spawn adds Arenas: 15-minute MOBA team fights — majidmanzarpour · 2026-10-20
- esp32sim-mcp: MCP server lets AI agents build and test ESP32 firmware in a local emulator — EmanueleStrazzullo · 2026-10-20
- GameSummoner one-line local install: prompt-to-game pipeline runs fully local on a 5090 — demonicpigg · 2026-10-20