Model grafting turns Qwen3.5-4B into a causal encoder-decoder, 3.7x faster long prompts
asankhs · reddit · 2026-09-22
Reddit user asankhs demonstrates "model grafting": inspired by DeepSeek-V4.1-Flash's causal encoder-decoder architecture (trained from scratch), the technique retrofits an existing pretrained model—cutting at some depth, letting lower layers read the prompt, using upper layers as prefix KV for the encoder via identity-init adapters, then healing with self-distillation from the unmodified parent. Decoding stays unchanged.
Applied to Qwen3.5-4B, two variants are released on Hugging Face (LocalLLaMA/Qwen3.5-4B-graft8 and graft16):
- graft8: 3.7x speedup at 128K prompts with some accuracy loss
- graft16: 2.0x prompt-processing speedup with minimal accuracy loss
A blog post details the method, offering an open-source path to faster long-context inference without training from scratch.
Related event: Model Grafting Converts Qwen3.5-4B into Causal Encoder-Decoder, 3.7x Faster(2 posts)→
More from Models
- How to cut Fable 5.2 token costs by 90%: Arc routing plus a lean CLAUDE.md — EXM7777 · 2026-11-12
- Partial RuneBench results: Inkling 2 Small scores 5.314, far above original — mariofilhoml · 2026-11-12
- Grok Bot 1.0 Announced: Shareable Bots With Upgraded Collaboration — SpaceXSI · 2026-11-12
- Polymarket odds put Chinese AI lab at #1 by year-end at just 6% — Polymarket · 2026-11-12
- FutureHouse's deslopper trains Kosmos to write indistinguishably from human experts — proteinrosh · 2026-11-12
- Developers say Gemini 4 Carbon is notably better at favicon design and micro-interactions — AI_Andrew · 2026-11-12