Model grafting turns Qwen3.5-4B into a causal encoder-decoder, 3.7x faster at 128K prompts
asankhs · reddit · 2026-09-22
A Reddit user demonstrates "model grafting": inspired by DeepSeek-V4.1-Flash's from-scratch causal encoder-decoder architecture, the method retrofits an existing model — cutting at some depth, letting lower layers read the prompt, and injecting upper-layer residual streams as prefix KV via identity-init adapters, then healing with self-distillation from the unmodified parent. The decoder stays unchanged. Details in the author's blog post.
Two variants of Qwen3.5-4B are released on Hugging Face:
- graft8: 3.7x prompt-processing speedup at 128K length, with some accuracy loss.
- graft16: accuracy very close to the parent model, 2.0x speedup.
A reproducible community experiment showing long-context efficiency gains can be added to pretrained models after the fact, with both variants open-sourced.
More from Research
- HWREBench: AI researcher hacks Amazon smart devices daily to benchmark hardware reverse engineering — johnowhitaker · 2026-09-22
- TMLR submissions quadruple on AI-generated influx; ICLR 2027 caps single authors — petitegeek · 2026-09-22
- Stanford's VirtualBiotech puts tens of thousands of AI scientist agents in Science, NYT reports — StanfordAILab · 2026-09-22
- Irit Dinur wins Gödel Prize for her landmark 2005 proof of the PCP theorem — willcb · 2026-09-22
- Researcher proposes using RL to teach AI when to give up — sqcai · 2026-09-22
- Nature publishes Delphy: scalable near-real-time Bayesian phylogenetics for outbreak tracking — burny_tech · 2026-09-22