Token superposition pretraining idea: prior experiments were 'pretty much a disaster'
giffmana · x · 2026-09-19
- Aleksa Gordic proposes TST: during early pretraining, group every s neighboring tokens and average their embeddings into one latent token that predicts the next non-overlapping bag of s tokens (inspired partly by resolution in vision).
- Kevin (giffmana) replies he ran related experiments inspired by visual resolution — most turned out to be "pretty much a disaster".
- A rare honest negative-result datapoint for multi-token training ideas.
Related event: Token Superposition Training Could Cut MoE Pretraining Compute by 2.5x(2 posts)→
More from Research
- Cambridge's van der Schaar Lab opens PhD applications for 2027 entry — MihaelaVDS · 2026-11-04
- Tencent's Hyra 2.0 Generates Protein Sequences 57x Faster Than ProteinMPNN — YvesMulkers · 2026-11-04
- Researcher Urges Top AI Conferences to Desk-Reject AI-Generated Papers Amid Fake Data Scandals — Vegetable-Formal-753 · 2026-11-04
- Harvard Professors Admit AI Article Reviews Now Beat the Best Humans — iamfakhrealam · 2026-11-04
- Pre-pretraining gains traced to embedding init: N(0,0.02) alone lifts perplexity 9% — LauraRuis · 2026-11-04
- kalomaze shares loss curve of larger-backbone run on DeepSeek V3.1 base, eval loss down to ~0.82 — kalomaze · 2026-11-04