Three probes track post-training SWE-bench rankings, BPB hits ρ=0.964
hyunw_kim · x · 2026-10-20
From the NVIDIA paper thread: across 10 base checkpoints, three probes track post-trained SWE-bench Verified rankings — Decisive-Action BPB at ρ=0.964, Prefix pass@32 at ρ=0.951, Patch MCQ at ρ=0.867. The trick: find the decisive step in successful trajectories (where the cumulative patch first flips failing tests to passing), then measure the base model's hidden potential with that context — no need to run each base model as a full coding agent.
More from coding & agent
- Microsoft open-sources Scope to evaluate how AI coding agents use your SDKs — DanWahlin · 2026-10-20
- Descript adds one-click import for Claude Motion animations, turning them into narrated videos — descript · 2026-10-20
- Dev builds a game demo in 2 days with Opus 5.5, Blender and Tripo assets — techartist_ · 2026-10-20
- Even Opus 5.5 hasn't internalized the bitter lesson when writing LLM systems — brandon_galang · 2026-10-20
- After 400+ agents, an engineer maps 15 AI agent memory patterns from types to control — MaryamMiradi · 2026-10-20
- Hackers hit AI coding tool OpenCode, stealing users' API keys — TechCrunch AI · 2026-10-20