Gradient clipping stress test: unclipped RNN diverges in 17 steps, clipped runs converge to 0.043 loss
cneuralnetwork · x · 2026-10-24
An experiment training an intentionally unstable RNN in PyTorch with identical weights, data, and learning rate across 5 seeds, comparing gradient clipping configurations:
- No clipping: all 5 runs diverged within 17 steps
- Clip norm 1.0: all runs completed 320 steps, median validation loss 0.043
- Clip 5.0: far less effective, with visibly higher validation loss
- Lowering the learning rate to 0.005 also prevented divergence without clipping
Mathematically, clipping rescales gradients via gclipped = g × min(1, c / ||g||₂), capping magnitude without changing direction; 94.7% of steps triggered clipping at threshold 1.0. Key takeaway: controlling parameter update magnitude alone can completely change training dynamics, and a loose clipping threshold (5.0) leaves performance on the table.
More from Research
- Kornia 0.9.0 ships a conventions page targeting LLM coding pitfalls — kornia_foss · 2026-10-24
- Fleuret: teacher forcing turns time iteration into depth iteration, and GPUs love it — francoisfleuret · 2026-10-24
- Domingos: An attention head is just a fast-weight MLP — pmddomingos · 2026-10-24
- Ross Taylor: studying LLM behavior will increasingly resemble social science — rosstaylor90 · 2026-10-24
- Ross Taylor on RLCR: simplicity is the final achievement, details matter — rosstaylor90 · 2026-10-24
- SenseTime open-sources 27B robot model matching GPT-6-Astra at ~1/50 the cost — rohanpaul_ai · 2026-10-24