Token Superposition Training Cuts Pretraining Compute 2.5x on 10B MoE
gordic_aleksa · x · 2026-09-19
Aleksa Gordić breaks down Nous Research's token superposition training (TST): during early pretraining, group every s neighboring tokens, average their embeddings into one latent token, and use it to predict the next non-overlapping bag of s tokens. This cuts sequence positions 4x, exposing the model to more raw text per FLOP, before switching back to standard next-token training to recover exact ordering. Their 10B MoE experiment reaches comparable/better loss with 2.5x less compute. Intuition: a blurred, low-resolution version of text first, then sharpening — akin to a Laplacian pyramid. Big caveat: it assumes a compute-bound, not data-bound regime, which contradicts today's consensus.
Related event: Token Superposition Training Could Cut MoE Pretraining Compute by 2.5x(2 posts)→
More from Infra
- Fire at Taiwan Panasonic PCB materials plant may tighten CCL supply further — zephyr_z9 · 2026-11-04
- Same Claude model served by Bedrock one day and Anthropic the next, dev finds — georgos_pappas · 2026-11-04
- Self-improving software is inevitable: closing the agent-observability loop — bibryam · 2026-11-04
- Gigabyte's AI TOP motherboard packs 2TB RAM and 9 GPUs for home AI labs — vincent_koc · 2026-11-04
- Building a 7-GPU 132GB VRAM local inference rig: consolidate on EPYC or split across two systems? — infectiousstupidity · 2026-11-04
- imec neural probe integrates photonics directly into 130nm CMOS backend — jwt0625 · 2026-11-04