Analysis: Qwen Engram follows Zipf distribution, prunes well
I-am_Sleepy · reddit · 2026-08-28
A user analyzed the Engram component (51B params) in Qwen 3.8 Next based on 40M tokens. It follows a strict Zipf distribution: the top 1% of rows handle 74% of accesses, and 76% of rows are untouched. Low-rank factorization and head pruning failed, but simple frequency pruning (keeping 50%) worked well, though cross-domain generalization remains poor.
More from Research
- DeepPhylo infers phylogenetic trees without unrealistic evolutionary assumptions — anshulkundaje · 2026-10-15
- Wander preprint explores graph foundation models via shared objective and random-walk interface — mmbronstein · 2026-10-15
- CHI Reviewer: AI Slop Is Real, but Unusually Rigorous Papers Still Getting Through — IanArawjo · 2026-10-15
- Into the Unknown: CMU benchmark finds 114 of 135 AI-written research papers fail integrity checks — arankomatsuzaki · 2026-10-15
- 8 non-negotiable principles for building AI that actually learns from experience — tokenbender · 2026-10-15
- Benchling launches Science Foundries: multiplayer agent swarms for end-to-end science workflows — nlarusstone · 2026-10-15