Sharing a Complete NanoGPT Speedrun
willdepue · x · 2026-07-14
The author shares a complete NanoGPT speedrun, focusing heavily on training and optimization techniques.
The process turned out to be more demanding than expected, utilizing the permutation trick. It also touches on the rivalry between Muon and duality-based optimizers, ultimately completing the challenge using these optimization strategies.
More from Research
- Chinese AI labs are now treating distillation obfuscation as the top research topic — pmddomingos · 2026-07-22
- Structural ensembles beat single predictions in TCR:pMHC generalization study — quaidmorris · 2026-07-22
- RSS launches under OMSF to push structural biology data modeling at scale — MoAlQuraishi · 2026-07-22
- enFoldX tops 8 neoantigen scans and an unseen-peptide benchmark — quaidmorris · 2026-07-22
- enFoldX reaches AUC 0.82 on human VDJdb and transfers to mouse at 0.76 — quaidmorris · 2026-07-22
- enFoldX gains accuracy as AF3 ensemble disagreement rises for non-binders — quaidmorris · 2026-07-22