Early Codex v1 scratchpad explores a power-decay learning-rate schedule
eliebakouch · x · 2026-07-21
A screenshot from the v1/codex/scratchpad shows an early Codex v1 idea note for an inverse-sqrt power-decay learning-rate schedule for NorMuon.
The note says the scratchpad contains a mix of ideas such as WSD, OneCycle, schedule-free NorMuon, D-Adaptation/Prodigy, and many small horizon/cooldown tweaks, but no standalone inverse-sqrt or power-law decay screen yet. It then explains why inverse-square-root and power-law schedules are common in transformer training, and suggests replacing the post-warmup global eta with a power decay:
- eta(t) = max(etamin, (1 + (t - warmupsteps) / tau) (-power))
More from coding & agent
- An MCP server signs every AI agent tool call into a verifiable Merkle chain — Funky_Chicken_22 · 2026-07-22
- Annotated transcript of a Claude Code team interview is now available — trq212 · 2026-07-22
- Claude Code skill uses 10 Markdown rules to make outputs ADHD-friendly — alex_verem · 2026-07-22
- BUZZ launches as an open-source group chat layer for teams and agents — Scobleizer · 2026-07-22
- A Firecracker-based platform says it can host 6,000 AI agents on one 256 GB server — maritime_sh · 2026-07-22
- A better path to agent autonomy is running waves, finding friction, and iterating — JnBrymn · 2026-07-22