ByteDance Seed: Effective Learning Rate Trajectories, Not Normalization, Drive WN Training Gains
ByteDance-Seed · hf · 2026-10-30
A paper from ByteDance Seed and Peking University shows that the recent pretraining loss gains attributed to weight-normalized (WN) training are driven primarily by the induced effective learning rate (ELR) trajectory rather than normalization alone — matching ELR trajectories largely collapses the training loss dynamics of WN and weight decay.
Schedule design should therefore target the ELR rather than the nominal learning rate. Building on this principle, the authors propose AIR (Affine Inverse-square-Root decay), which improves both large-scale pretraining loss and robustness.
More from Infra
- Ollama installed ≠ model downloaded: pitfalls when running it on cloud GPUs — NoConflict2854 · 2026-10-30
- NVIDIA launches CUDA Rust: two tracks for writing GPU kernels in Rust — AxSaucedo · 2026-10-30
- Rapid-H3 renders MiniMax-H3 video+audio 3-5x faster on one DGX Spark — vchauhan_ · 2026-10-30
- KAIST's LOWN Cuts Diffusion VLM Decoding FLOPs to 35% While Keeping 99.6% Accuracy — kaist-ai · 2026-10-30
- Dell workstation with 96GB NVIDIA GPU may cost less than the card alone — chemist_slime · 2026-10-30
- Midwit meme on Databricks vs Snowflake: just put it in S3 — cto_junior · 2026-10-30