NanoGPT embedding table gains validated by earlier ngrammer paper, both using AdaGrad
_arohan_ · x · 2026-09-29
arohan highlighted striking results from an embedding table experiment on NanoGPT, noting the earlier ngrammer paper reported similar findings and that both approaches rely on AdaGrad. He calls it a free win — a solid improvement with no extra cost, showing ideas from the literature translating directly into open-source reproduction.
Related event: NanoGPT Embedding Table Gains Echo ngrammer Paper Findings(2 posts)→
More from Models
- Researcher suggests Claude's 'claudish' gibberish may signal Opus communicating beyond human comprehension — JeremyNguyenPhD · 2026-09-29
- Design challenge bonus: Meta Muse almost works but breaks completely on page 3 — NathanWilbanks_ · 2026-09-29
- Late-layer neurons in Qwen act like on-off switches, unlike Olmo — Sauers_ · 2026-09-29
- After a day of real use: Sonnet 5.5 is not just Opus 5.5 at half price — alexcovo_eth · 2026-09-29
- Holo4-27B post-train cuts Qwen3.8-27B agent tokens by 79% on same task — solyarisoftware · 2026-09-29
- ProgramBench: Best Model Solves Only 4.5% of Binary-to-Code Rebuild Tasks — jyangballin · 2026-09-29