What it takes to squeeze performance: EMA, cosine scheduling and custom dead-cluster revival
chrisemoody · x · 2026-11-11
Replying to how their run differs, swannbessa lists the stacked optimization components: EMA updates, cosine scheduling on the step size, scaling laws for hyperparameters, a custom dead-cluster revival mechanism, plus initialization and finalization details.
More from Research
- Sakana AI × Pathway Post-Transformers lecture on AI and the brain now online — kaixhin · 2026-11-11
- Defending METR: public data can't second-guess its AI speedup findings — peterwildeford · 2026-11-11
- Adversarial RL envs for backdoor insertion vs. detection: where does it converge? — DimitrisPapail · 2026-11-11
- UniPath (NeurIPS 2026): adaptively coordinating understanding and generation in one model — jindong_wang92 · 2026-11-11
- Versor pitches real2sim2real: robotics sim needs physical dynamics, not just visuals — abhishekunique7 · 2026-11-11
- Scale AI's 1,200-run study: PPO wins at small batches, GRPO catches up at scale — heghbalz · 2026-11-11