Token Superposition Training Cuts Pretraining Compute 2.5x on 10B MoE
gordic_aleksa · x · 2026-09-19
Aleksa Gordić breaks down Nous Research's token superposition training (TST): during early pretraining, group every s neighboring tokens, average their embeddings into one latent token, and use it to predict the next non-overlapping bag of s tokens. This cuts sequence positions 4x, exposing the model to more raw text per FLOP, before switching back to standard next-token training to recover exact ordering. Their 10B MoE experiment reaches comparable/better loss with 2.5x less compute. Intuition: a blurred, low-resolution version of text first, then sharpening — akin to a Laplacian pyramid. Big caveat: it assumes a compute-bound, not data-bound regime, which contradicts today's consensus.
Related event: Token Superposition Training Could Cut MoE Pretraining Compute by 2.5x(2 posts)→
More from Infra
- Polymarket puts 23% odds on an orbital AI data center launching by end of 2027 — Polymarket · 2026-09-19
- US chip fabs face 157,000-worker shortfall as only 3% of engineering grads enter chipmaking — ns123abc · 2026-09-19
- Phylo cuts AI inference cost 60% with open-weight models on Fireworks as usage doubles monthly — sophiamyang · 2026-09-19
- Only 3% of US engineering students go into chips as AI lures the rest — ns123abc · 2026-09-19
- Cerebras launches Money Agent, a finance assistant powered by Qwen3 27B — irinarish · 2026-09-19
- Cloudflare saved another 100TB of RAM by reworking consistent hashing in Rust — Cloudflare Blog · 2026-09-19