ADANA paper thread part 2: caveats and confirmation of exponent-level optimizer gains
_katieeverett · x · 2026-09-10
Katie Everett wraps up part 2 of her ADANA optimizer paper thread, noting three caveats: small models (51M–253M), fixed batch of 256×2048 (ADANA may lose efficiency at larger batch), and some points requiring extrapolated baseline fits. She emphasizes this is the first time she's convinced optimizers can genuinely improve scaling exponents in Transformers, with more experiments forthcoming.
More from Research
- Token Collapse: Study Finds Graph Soft Prompting Tokens in LLMs Fail to Encode Structure — Sapienza · 2026-10-27
- Mathathon 2.0 launches with Terence Tao's SAIR, $1.5M in compute for teams — _sathvikr · 2026-10-27
- The 'sweet lesson' is real: two kinds of ML problems need opposite strategies — mike64_t · 2026-10-27
- PixGR: Pixel-space generative decoding for fast feed-forward 3D reconstruction — kwangmoo_yi · 2026-10-27
- UniAction: unified action conditioning lets one world model handle mixed action types — ZeYanjie · 2026-10-27
- Infini-AI Lab open-sources Hermit, a framework for agent harness evolution and continual learning — BeidiChen · 2026-10-27