LLM pretraining's next step: bringing data into architecture-optimizer co-design
akbirthko · x · 2026-11-04
The author frames industrial LLM pretraining around three interlocking circles: architecture, data, and optimization. Co-design of architecture and optimization has advanced via custom optimizers for embeddings, LM heads, modules like Over-Encoding and Engram, and Muon for selected Transformer weight matrices. The next step is folding data into the co-design — e.g., jointly designing data schedules with learning-rate schedules — noting n-gram modules' training dynamics already depend on data.
More from Research
- RoboColosseum launches open robot eval with 100+ real robots and 10,000 free eval hours — DJiafei · 2026-11-04
- NVIDIA releases Dory1.0-4B, a looped recurrent LM beating larger models — nvidia · 2026-11-04
- Running controlled LLM experiments with Codex CLI: disable every tool via config — lateinteraction · 2026-11-04
- Claude Opus 5 surgically edits ~1% of ProteinMPNN weights into SolubleMPNN-like behavior — AllThingsApx · 2026-11-04
- ESMFold author: AlphaFold-style covariation could unlock small-molecule prediction — sokrypton · 2026-11-04
- Michael Levin publishes interactive concept atlas of his mind, life & agency research — drmichaellevin · 2026-11-04