Same GRPO recipe on three from-scratch LLMs yields wildly different results with no clean scale relationship
john_enev · reddit · 2026-08-20
The author trained three LLMs from scratch in raw PyTorch (353M/316M/672M), then applied SFT + GRPO to each with identical curriculum, reward, hyperparameters, and KL coefficient:
- Pre-training: val loss improved with modernization and scale (2.87 → 2.78 → 2.59).
- After SFT: WikiText perplexity rose for all three (expected at this scale), with smaller degradation for bigger models.
- After GRPO: V1 barely moved (+0.2%), V2 exploded +52%, V3 only +5%—no clean pattern by scale; downstream tasks moved the same way (arceasy dropped 6 points on V3). The models learned the curriculum but nothing transferred: GSM8K stayed near zero, and with no length penalty they often wouldn't stop generating.
- Confounds acknowledged: V2→V3 changed params, tokens, data mix, and attention (DiffAttn→XSA) simultaneously; GRPO used a bare solver template vs. SFT's chat format; no stop reward; early curriculum stages never re-evaluated. Total cost $750, so no ablations.
- Inference: a hand-written GQA-aware per-request KV cache matched full-forward logits to 1.4e-06; 100-token generation sped up 3.7x at 32-prompt to 10.1x at 512.
All nine checkpoints and a comparison playground are open on Hugging Face.
More from Models
- ChatGPT's citations drop across all social media domains, not just Reddit — _AustinCalvert_ · 2026-08-20
- Luna leads cost-efficiency, Qwen 27B stands out as laptop-viable option — MikePFrank · 2026-08-20
- Speculation: Opus 5 issues caused by Mythos RLAIF — rickasaurus · 2026-08-20
- Generalist AI Releases GEN-1.5 One-Shot Learning Model — GraceToSentience · 2026-08-20
- llama.cpp adds support for new Granite SWA and MoE models — jacek2023 · 2026-08-20
- Nvidia prioritizes Nemotron open-source models to rival top global models — pstAsiatech · 2026-08-20