COUNT Paper: N-gram Counts on Host Memory Equal 57% More Training Compute
pmttyji · reddit · 2026-10-20
The COUNT paper stores precomputed n-gram count tables on cheap host memory and builds n-gram embeddings by weighting the LM's own unembeddings with corpus frequencies — no new learned parameters needed.
Key results
- At a fixed compute budget, adding COUNT to 1B-parameter LMs improves validation loss as much as 57% more training compute.
- When GPU memory is limited, keeping backbone parameters on GPU with COUNT on host beats splitting memory with learned n-gram embeddings.
- Swap in a domain-specific n-gram count table at inference time to adapt to new tasks with zero training.
Code, paper, and models are available on arXiv, GitHub, and HuggingFace.
More from Infra
- Mac Studio with 256GB RAM now shipping in 15-17 weeks as demand outstrips supply — LinusEkenstam · 2026-10-20
- Google open-sources TorchTPU to run PyTorch workloads on TPUs with minimal code changes — rseroter · 2026-10-20
- Simular's neuro-symbolic agent runs 217x cheaper than Opus at 72% on OSWorld — xwang_lk · 2026-10-20
- Ex-Google expert: data center capacity and electricity, not GPUs, are the real AI bottleneck — RihardJarc · 2026-10-20
- a16z hosts AI Infra Day with fal, Black Forest Labs, Krea and ElevenLabs engineers — venturetwins · 2026-10-20
- Video generation research runs on AMD MI350X, PRs merged into PyTorch nightly — vincesitzmann · 2026-10-20