Chorus: Shared N-gram Embedding Tables Cut Table Size 87% While Beating Per-Layer Designs

Stephen Chung · hf · 2026-11-19

Stephen Chung and Wenyu Du of DualverseAI propose Chorus, a new take on n-gram embedding tables. Prior designs give each layer its own table — costly in memory and blocking knowledge sharing — while naively sharing one table starves layers of learnable preferences and underperforms.

Chorus shares a single set of n-gram tables across the whole network; each layer rescales embedding components by factors computed from its current hidden state and reads them through its own projection.

Results: with 1.2B table parameters, Chorus predicts text better than FactorEmbed and DeepSeek's Engram do with 9.6B — an 87% smaller table — and outperforms Engram, TN-gram and FactorEngram across table sizes from 20M to 9.6B. Paired with the memory-efficient optimizer Ember, it saves 99.2% of table-optimizer memory vs AdamW without degrading training.

Original post →

More from Infra

Infra channel →