NVIDIA paper predicts coding-agent potential of base models before post-training

On October 20, a team led by NVIDIA researcher Hyunwoo Kim released the paper "Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Models" (arXiv:2610.10…). The core claim: there's no need to run full post-training on every base model to find the best one — with the right observables, a base model's coding-agent potential can be predicted before post-training.

Confirmed

• The team's method replays successful agent trajectories to locate the "decisive step T" — the first FAIL→PASS flip — and probes the base model with three tests using real harness context (test task, trajectory prefix, and environment state at T)
• Across 10 base models, the three probes track post-training SWE-bench Verified rankings with correlations of Decisive-Action BPB ρ=0.964 and Prefix pass@32 ρ=0.951 (the third probe was truncated in the post)
• The DeepSWE-derived decisive-action probe tracks SWE-bench Multilingual at ρ=0.976, and prefix pass@32 was also validated on Terminal-Bench (specific value truncated in the post)
• Traditional benchmarks predict poorly: HumanEval shows negative correlation (ρ=-0.394), RepoBench XFirst ρ=0.830
• The paper also exposes a reality: 5 of 6 base models score 0% on SWE-bench Verified — base models often can't even operate a coding-agent harness — while their post-trained versions reach up to 80.6%

Why it matters

• Post-training is expensive; if every candidate base model had to be fully trained before comparison, the compute and time cost would be enormous — this work offers a shortcut for base model screening
• The results also show that traditional static coding benchmarks (like HumanEval) are disconnected from real agent capability; the negative correlation is a warning for industry model-selection conventions

2026-10-20 ~ 2026-10-20 · 8 related posts

• NVIDIA asks: must you post-train every base model to find the best coding agent? — hyunw_kim · 2026-10-20
• [source] 5 of 6 base models score 0% on SWE-bench, yet hit 80.6% after post-training — hyunw_kim · 2026-10-20
• Visual explainer: three probes rank base models via the decisive trajectory step — hyunw_kim · 2026-10-20
• [source] Three probes track post-training SWE-bench rankings, BPB hits ρ=0.964 — hyunw_kim · 2026-10-20
• HumanEval predicts coding-agent potential negatively; new probe hits ρ=0.964 — hyunw_kim · 2026-10-20
• Base-model probes predict cross-benchmark coding-agent potential at ρ=0.976 — hyunw_kim · 2026-10-20
• [source] NVIDIA paper predicts post-training coding-agent potential from base models at ρ=0.964 — hyunw_kim · 2026-10-20
• Paper: 'Before They Can Solve' predicts a base model's post-training potential early — hyunw_kim · 2026-10-20