Prune-then-loop retrofit turns a pretrained VLM recurrent, beating baseline with 23.3% fewer language params

HildeKuehne · x · 2026-10-31

Hilde Kühne's team published a new paper, Retrofitting Recurrent Vision-Language Models from Pretrained Backbones, introducing a prune-then-loop strategy that converts an off-the-shelf pretrained VLM into a depth-recurrent model.

Core method

Prune a subset of layers from the VLM's language backbone and loop over the remaining layers
A linear adapter at each loop step merges the block's output with prior token representations and projects it back

Three-stage recovery training

Text-only pretraining to repair the language backbone
Image-text training to restore projector alignment
Full visual instruction fine-tuning

Distillation from the original non-recurrent model is applied in the first and last stages.

Results (validated on TinyLLaVA-1.4B, no new vision-language data)

The retrofitted model stores 23.3% fewer language parameters and scores 58.13 average, beating the unpruned baseline trained with the same recovery data (55.80)
Gains concentrate on reasoning-oriented benchmarks: SEED, AI2D, CV-Bench
Ablations show improvements come from recurrence itself rather than recovery training; block-wise recurrence beats layer looping, the adapter and distillation both matter, and training with variable recurrence makes the model robust to different inference-time depths.

Original post →

Prune-then-loop retrofit turns a pretrained VLM recurrent, beating baseline with 23.3% fewer language params

More from Research