NVIDIA open-sources midtraining recipe: 500B tokens lift Nemotron agentic benchmark scores
denisyarats · x · 2026-10-17
NVIDIA published an open midtraining recipe that primes base models for agentic post-training, with Perplexity contributing agent traces. The stage trains at 256K context on hundreds of billions of tokens mixing packed repos, PRs, commit histories and build logs with a small share of agentic traces.
- Nemotron 3.5 Lightning: 500B midtraining tokens raise SWE-bench Verified 60.7→64.5, Multilingual 55.8→62.8, Pro 32.0→35.5 after identical SFT
- Nemotron 3.5 Super: 280B tokens lift DeepSWE 11.7→15.7 and SWE-bench Verified 72.9→74.3
The paper argues pretraining lacks agent interactions while overly long SFT hurts later RL trainability, and details data sourcing, ablations and eval protocol.
More from Models
- Opus 5.5 draws stick figures to nail camera composition, wowing users — justin_hart · 2026-10-17
- Meta drops 816B open-weight MoE, Anthropic hits $16.8B Q3, 8 states sue OpenAI — rohanpaul_ai · 2026-10-17
- Andriy Burkov: the only gap between open-weight and frontier models is data quality — burkov · 2026-10-17
- Quick take: Claude 5.5 Haiku trails 6.1 Luna, which roughly matches Gemini 3.8 Flash — burkov · 2026-10-17
- Anthropic launches Claude Haiku 5.5: 75% cheaper, 39.2% on Terminal-Bench vs GPT-6 Luna's 16.4% — dl_weekly · 2026-10-17
- OpenAI boosts effective Codex usage via context reuse and better cache hits — TheMoonMidas · 2026-10-17