797GB n-gram Table in CPU Memory Boosts Pretraining Compute by 57%

Ian Magnusson's COUNT project stores a precomputed 797GB n-gram count table in read-only CPU memory to aid language model training, matching the results of using 57% more compute at the 1B-parameter scale. A similar trick was recently used to break nanoGPT speedrun records.

2026-10-15 ~ 2026-10-16 · 2 related posts

• 797GB of n-gram counts in CPU RAM let 1B models match 57% more compute — PangWeiKoh · 2026-10-15
• 797GB of n-grams in read-only CPU RAM match 57% more compute at 1B-scale LM pretraining — anshulkundaje · 2026-10-16