Chorus paper: shared n-gram tables cut memory 87% while beating DeepSeek's Engram
brown2green · reddit · 2026-11-20
A new arXiv paper proposes Chorus, a shared n-gram embedding design that gives language models extra capacity at minimal compute cost.
- Existing designs (like DeepSeek's Engram) use a separate n-gram table per layer, which is memory-hungry and blocks cross-layer knowledge sharing; a naive single shared table underperforms.
- Chorus uses one set of tables for the whole network; each layer reads it via its own projection, rescaling components by factors from its hidden state.
- Results: tables are 87% smaller — a 1.2B-parameter table beats FactorEngram and DeepSeek's Engram at 9.6B on text prediction, winning across table sizes from 20M to 9.6B.
- Paired with the Ember optimizer, it saves 99.2% of table-optimizer memory vs AdamW without degrading training.
The poster notes Engram tables can scale well beyond backbone size with performance tracking table size, dispelling the idea that only 25% of weights can be Engram layers.
Related event: Chorus: Shared n-gram Embedding Table Shrinks 87% While Quality Rises(2 posts)→
More from Infra
- Quantum Computing Takes Aim at AI Workloads as IBM and QuEra Map Fault-Tolerant Roadmaps — mikeflache · 2026-11-20
- Tencent Earnings Call: Latest Chip Restrictions Will Accelerate China's Domestic Semiconductor Supply Chain — basedjensen · 2026-11-20
- px0: A 20MB browser-based IDE for AI-generated code, built by chopping layers with first-principles thinking — arpit_bhayani · 2026-11-20
- Tsinghua-spinoff Qingyi launches Ascend 950DT 10,000-GPU token factory, lifting utilization from 40% to 80% — 量子位 · 2026-11-20
- SenseNova-Loop 9B runs on 8GB VRAM at 22 t/s, uses half the tokens of Qwen3.8 — R_Duncan · 2026-11-20
- DeepSeek job posting reveals 100k-accelerator cluster challenges amid Ascen 950 training rumor — teortaxesTex · 2026-11-20