UW's COUNT stores n-gram counts on host memory, giving 1B LMs the equivalent of 57% more compute
UW · hf · 2026-10-15
University of researchers introduce COUNT (Counting Offline Unlocks N-gram Transformers), which scales language model memory with n-gram embeddings without training new parameters — instead storing precomputed n-gram counts on cheap host memory.
- Mechanism: corpus-frequency-weighted averages of the LM's own unembeddings for the tokens most often following the current n-gram history, so the table is read-only and avoids costly GPU memory training.
- At a fixed compute budget, adding COUNT embeddings to pretraining 1B-parameter LMs improves validation loss as much as 57% more compute.
- When GPU memory is limited, letting the backbone fill GPU memory with COUNT on host memory beats splitting memory with learned n-gram embeddings.
- Bonus: swapping in a domain-specific n-gram count table at inference adapts to new tasks with zero training.
More from Infra
- vLLM P/D prototype hits 91.24% prefix cache hit rate on 12× GB300 — vllm_project · 2026-10-15
- vLLM ships Day-0 support for Step 5 Preview, cuts sparse prefill latency 2.2% on B300 — vllm_project · 2026-10-15
- vLLM details Step 5 Preview's region-based sparse attention: score summaries, then attend — vllm_project · 2026-10-15
- vLLM announces Day-0 support for Step 5 Preview across Blackwell and Hopper — vllm_project · 2026-10-15
- Musk amplifies projection: 5,000 Starship launches a year moving 1M tonnes to orbit — elonmusk · 2026-10-15
- SpaceX seeks FAA nod for 456 Starship launches/year; Raymond James sets $800 street-high target — elonmusk · 2026-10-15