Model grafting turns Qwen3.5-4B into a causal encoder-decoder, 3.7x faster long prompts
asankhs · reddit · 2026-09-22
Reddit user asankhs demonstrates "model grafting": inspired by DeepSeek-V4.1-Flash's causal encoder-decoder architecture (trained from scratch), the technique retrofits an existing pretrained model—cutting at some depth, letting lower layers read the prompt, using upper layers as prefix KV for the encoder via identity-init adapters, then healing with self-distillation from the unmodified parent. Decoding stays unchanged.
Applied to Qwen3.5-4B, two variants are released on Hugging Face (LocalLLaMA/Qwen3.5-4B-graft8 and graft16):
- graft8: 3.7x speedup at 128K prompts with some accuracy loss
- graft16: 2.0x prompt-processing speedup with minimal accuracy loss
A blog post details the method, offering an open-source path to faster long-context inference without training from scratch.
Related event: Model Grafting Converts Qwen3.5-4B into Causal Encoder-Decoder, 3.7x Faster(2 posts)→
More from Models
- MiMo-V2.6 Flash and Pro Both Pass a 100-Step Clinical Agent Workflow That V2.5-Pro Failed — MaziyarPanahi · 2026-09-22
- GPT-6 Astra tops Claude Opus 5 in Ramp business spending share, OpenAI overtakes Anthropic — firstadopter · 2026-09-22
- 86 physics questions, 100 runs each: model hits 87.6% accuracy but understates confidence — Ok-Challenge-7810 · 2026-09-22
- DeepSeek reportedly bets on Huawei chips to train next-gen models; Liang says it 'has to work' — kimmonismus · 2026-09-22
- Xiaomi's MiMo-V2.6-Pro tops open models on $2.62M RL; Anthropic alleges Claude distillation — The Decoder · 2026-09-22
- Tencent finally opens WeChat interface, unlocking 100GB+ chat data processing — Xianbao_QIAN · 2026-09-22