5 of 6 base models score 0% on SWE-bench, yet hit 80.6% after post-training
hyunw_kim · x · 2026-10-20
The motivating problem from the NVIDIA thread: base models often can't even operate a coding-agent harness — 5 of 6 base models scored 0% on SWE-bench Verified, yet their post-trained versions reached up to 80.6%. Hence the decisive-step probe method: how to spot hidden potential when the benchmark shows a wall of zeros.
More from coding & agent
- Dev turns his Goodreads shelf into a slick site you can rebuild with Claude — thisiskp_ · 2026-10-20
- Notion AI ships major upgrade: smarter Agents, Auto 90% cheaper, BYOK support — ivanhzhao · 2026-10-20
- iPhone Action Button + Grok voice + HeyGen MCP turns a sentence into an avatar video — HeyGen · 2026-10-20
- Vibe-coded game League of Spawn adds Arenas: 15-minute MOBA team fights — majidmanzarpour · 2026-10-20
- esp32sim-mcp: MCP server lets AI agents build and test ESP32 firmware in a local emulator — EmanueleStrazzullo · 2026-10-20
- GameSummoner one-line local install: prompt-to-game pipeline runs fully local on a 5090 — demonicpigg · 2026-10-20