Transferring Qwen3.8's n-gram memory into a 0.8B model cuts perplexity 5.05%
Nicolodeva · reddit · 2026-09-25
A solo experiment, Qwengram-0.8B, transfers Qwen3.8-Flash-Next's 51B-parameter pretrained PLE n-gram memory into a frozen Qwen3.5-0.8B backbone, training only an R=1 reader at decoder layers 3 and 9 with a dynamic token-level gate. Validation perplexity drops from 18.28 to 17.35 (-5.05%) with no backbone fine-tuning. Key findings: real PLE beats random/permuted controls, 15M-token readers and R=1 are the sweet spot (20M/R=4 regress on math/code), and Q80 quantization retains 99.1% of the gain. Models, training code, and a modified llama.cpp runtime are open-sourced.
More from Models
- Micius-Flash: 27B Multimodal Model for Hardware and Semiconductor Agents — zhiman-ai · 2026-11-11
- Baidu says RSI auto-optimized ERNIE inference, 6.24x decode speedup on Kunxin P800 — xiaohu · 2026-11-11
- Baidu's ECHO: multi-token prediction reusing the last layer as a drafter — xiaohu · 2026-11-11
- ERNIE 5.5 Flash Preview: 280B-param multi-router MoE activating only 14B per token — xiaohu · 2026-11-11
- ERNIE 5.5 recreates the Pelican test: 117-second animation drawn frame-by-frame in Python — xiaohu · 2026-11-11
- 35,000+ Grok-built apps exposed patient data, plaintext passwords, and sales pipeline — The Decoder · 2026-11-11