StepFun blog details activation gradient clipping, up to 28% efficiency gain at scale
eliebakouch · x · 2026-10-16
HF researcher eliebakouch notes StepFun published a blog on activation gradient clipping similar to his team's. Clipping just 0.00004% of gradient values per layer suppresses gradient spikes, with gains growing with scale (+28% hold-out efficiency at L42). StepFun also used it as a diagnostic to trace a malformed training example back to a preprocessing bug.
More from Research
- Domingos Cites New Result Claiming RL Is 'Basically Useless' vs Supervised Learning — pmddomingos · 2026-10-23
- Wolfram Opens Rule 30 Prizes to AIs: $30,000 for Three Open Problems — cephaloform · 2026-10-23
- Stanford and Databricks use AI agents to reproduce medical findings and test analytical robustness — ChengleiSi · 2026-10-23
- EMNLP paper: diffusion LMs internally encode latent denoising timestep representations — gsarti_ · 2026-10-23
- KU Leuven and VIB open two Group Leader professorships in AI for Biology, start 2027 — jpillowtime · 2026-10-23
- Subspace Tuning curbs LLM catastrophic forgetting by freezing principal singular directions — scottniekum · 2026-10-23