CPU decode speed may depend on active weights per token, not total parameters

WildPino25 · reddit · 2026-07-29

A Reddit user proposes that on CPUs, decode speed depends on active parameters per token, not the model’s total parameter count.

• The post argues that if generation is limited by memory bandwidth, a model can scale total capacity without slowing inference, as long as only a small set of weights are active for each token.
• The author describes an architecture built around ternary weights, a granular MoE, and other sparsity tricks to keep active parameters small.
• Early measurements on a Ryzen 5 3600X reportedly raised throughput from 176 tok/s to 848 tok/s on an 8.3M sandbox model, with only a tiny quality cost.
• They also report a Kaggle training run on a 30M model and note that distillation from a bigger teacher underperformed plain cross-entropy by 0.0116 BPB, so they switched the recipe to CE-first.
• The author says anything beyond the sandbox is not yet trained, and the 10B model is still the target.

Original post →

More from Research

• ResearchArena tests whether monitors can catch sabotage in automated AI R&D — maksym_andr · 2026-07-29
• Benchmark one case, but rerun many tests to catch regressions — lemire · 2026-07-29
• A research talk compares SWE-bench, CodeClash, and ProgramBench for agentic coding — OfirPress · 2026-07-29
• Fluorescent Soybeans: Drone Hyperspectral Imaging Enables Early Disease Detection — NikoMcCarty · 2026-07-29
• Deconstructing scaling laws as a triad of optimization, architecture, and data — ChengleiSi · 2026-07-29
• Kuna debuts as an agent-driven Rust decompiler, ranking near Hex-Rays in structuring — moyix · 2026-07-29

Research channel →