ShellElastic trains one supernet yielding all sizes, with provable hyperparameter transfer
Can Jin · hf · 2026-10-14
ShellElastic organizes hidden dimensions into a dense core plus shells so every hidden matrix is block lower triangular: one full-width forward pass contains every sub-network's exact hidden states, keeping the compute graph fixed at roughly the cost of the widest sub-network. The paper also introduces μE, extending maximal update parameterization to elastic architectures, with provable guarantees that all sub-networks stay maximally updated simultaneously and that hyperparameters tuned on a small proxy transfer across sizes. One run yields sub-networks beating MatFormer and fully elastic baselines at matched parameter counts, approaching separately trained models that take several times the steps.
More from Infra
- Only 7% of frontier lab compute to pretraining by Q4 2026, but the viral chart is being misread — repligate · 2026-10-14
- AI agents hit monorepo scale: you need Google-style copy-on-write primitives, not Git — rickasaurus · 2026-10-14
- Modal trains RL models on mixed-accelerator clusters, matching homogeneous setups at lower cost — charles_irl · 2026-10-14
- A backyard "distributed data center" ADU startup for local LLM enthusiasts — dee_hw · 2026-10-14
- Dots Studio's DoSA speeds sparse attention indexer up to 2.74x over SGLang, open-sourced — dots-studio · 2026-10-14
- Report: Microsoft strains under data center spending and AI talent war, mulls cost cuts — SumitGup · 2026-10-14