Learned n-grams compress indexing to group tokens like chars into words

antirez · x · 2026-08-30

N-grams operate at a higher level similar to grouping characters into words (e.g., 'apple'). Since bi- and trigrams are too numerous, a compressing hashing is used for indexing.

Related event: antirez: Qwen 3.8 Flash Next's 51B N-gram Table Could Be Best Local Inference Option on 64GB Macs(7 posts)→

Original post →

More from Research

• ForestDiffusion: XGBoost-based tabular data diffusion model favors CPU parallelization — jm_alexia · 2026-08-30
• Accio Open-Sources CommerceAgentBench, Claude Passes 52% — dr_cintas · 2026-08-30
• Elbow Method: Evaluating Optimal Cluster Count — mdancho84 · 2026-08-30
• K-Means Iteration: Repeating Until Centroids Converge — mdancho84 · 2026-08-30
• Data Scientist Breaks Down K-means Clustering for Beginners — mdancho84 · 2026-08-30
• Code World Model: Coding Agent as World Brain — Charuru · 2026-08-30

Research channel →