Analysis: Qwen Engram follows Zipf distribution, prunes well
I-am_Sleepy · reddit · 2026-08-28
A user analyzed the Engram component (51B params) in Qwen 3.8 Next based on 40M tokens. It follows a strict Zipf distribution: the top 1% of rows handle 74% of accesses, and 76% of rows are untouched. Low-rank factorization and head pruning failed, but simple frequency pruning (keeping 50%) worked well, though cross-domain generalization remains poor.
More from Research
- MATM Framework: Multi-Agent Transactive Memory Enables Population-Level Trajectory Sharing — 841io · 2026-08-28
- On Information Sharing and Subtasks in Multi-Agent Systems — 841io · 2026-08-28
- Demystifying Singularities in ML Metrics: Infinite Horizon ≠ Infinite Work — tobyordoxford · 2026-08-28
- EMNLP 2026 Paper: Rethinking Multilingual Reasoning Gap with Layer Swap — IgorCarron · 2026-08-28
- Toby Ord: Intelligence measures may not saturate, could reach infinity — tobyordoxford · 2026-08-28
- ConforNets Enables State Transfer in AlphaFold3, SOTA on Conformations — jajoosam · 2026-08-28