Caveats: single-seed results, no clear win at 405M-1.4B evals
ziv_ravid · x · 2026-11-19
The author discloses limitations: mostly single-seed runs; no help for AdamW with ReLU² at 405M; downstream evals at 405M and 1.4B show no clear win over constant decay; and the reason this step-size trajectory works remains unexplained.
Related event: Temporary Early Weight Decay in Muon Pretraining Shows Delayed Payoff(8 posts)→
More from Research
- Shanghai AI Lab's GameRealm: 120M frames of gameplay data for visual foundation models — Shanghai-AI-Laboratory · 2026-11-19
- KradleBench launches as a dynamic multi-agent benchmark pitting LLMs, world models and humans against each other — lmoroney · 2026-11-19
- KradleBench Launches: Minecraft Legend SethBling Dethroned by Model Astra 6.2 — fchollet · 2026-11-19
- Transformer Lab unveils Primus, a 'society of researchers' of AI agents, and is hiring — carsonfarmer · 2026-11-19
- New LessWrong Analysis: LLMs Seek the Grader's Score, Not the Actual Goal — 1a3orn · 2026-11-19
- tabgen-eval: an open-source eval framework and benchmark for synthetic tabular data — LucaAmb · 2026-11-19