Recurrent retrofitting cuts VLM language params 23.3% while beating the unpruned baseline
HildeKuehne · x · 2026-10-31
An arXiv paper by Madhav Iyengar, Andreas Geiger and Hilde Kühne retrofits depth recurrence into pretrained vision-language models: prune layers from the language backbone and loop over a remaining subset, with a linear adapter fusing each block's output back into token representations. A three-stage recovery (language-only pretraining, vision-language projector alignment, visual instruction tuning) uses distillation from the original model in stages one and three. On TinyLLaVA-1.4B, with no new vision-language data, the retrofitted recurrent model stores 23.3% fewer language parameters yet outperforms the unpruned baseline trained on the same recovery data. Ablations: block-wise recurrence beats layer looping; the adapter matters; distillation helps; variable recurrence during training improves robustness to different inference-time iteration counts.
Related event: Prune-then-Loop Retrofitting Turns Pretrained VLMs into Recurrent Models(3 posts)→
More from Research
- Removing activation bias is a free trick that improves ViT, DiT and GPT training — ZimingLiu11 · 2026-10-31
- Hessian eigenvalue plots show removing activation bias flattens the loss landscape — ZimingLiu11 · 2026-10-31
- Zero-mean 'bug' in ReLU/SiLU/GeLU found to cause loss spikes; fixing it boosts LLMs and ViTs — ZimingLiu11 · 2026-10-31
- Biology Isn't Just a Data Problem: The lncRNA ANRIL Breaks Every Model You'd Write — iskander · 2026-10-31
- KAN boosts TabPFN scaling forecasts: OOD error cut up to 80% on StepLaw tasks — ZimingLiu11 · 2026-10-31
- Pre-registered training-curve forecast hits: TabPFN covers 14/14 later checkpoints — ZimingLiu11 · 2026-10-31