Boosting Visual Instruction Tuning with Self-Supervised Guidance
Sophia Sirko-Galouchenko, Monika Wysoczanska, Andrei Bursuc, Nicolas Thome, Spyros Gidaris
cs.CV
2026-04-15
Valeo.ai turns rotation, color matching, and correspondence into SFT instructions. Adding 3-10% of them lifts LLaVA vision-centric averages by about 1 point, +2.2 under LoRA.
LLaVA-style MLLMs can caption an image and still fail at counting lamps or judging left versus right. CLIP and DINOv2 already encode those cues. The leak sits in instruction tuning. Many image-instruction-response triples can be answered from language priors, so the decoder learns to ignore visual tokens.
Valeo.ai and Sorbonne treat this as modality competition. If the mix rarely punishes a text-only guess, the model keeps guessing from text. The vision encoder is not the missing piece. Use of its features is.
V-GIFT leaves the architecture, the loss, and the optimizer untouched. It only changes the data mix. Three classic SSL pretext tasks become ordinary instruction triples, trained with the same next-token objective.
The mix ratio ρ is SSL count divided by the original instruction set size. LLaVA-1.5 uses 10%. LLaVA-OneVision-1.5 uses 3%. Learning rate, batch size, and the original recipe stay fixed. Extra compute scales linearly with ρ: on 4×H100, full SFT of LLaVA-1.5-Qwen goes from about 12 hours to 14 hours at 110% data.
Averages over three seeds. Four vision-centric suites (CV-Bench 2D, POPE, MMStar, BLINK):
| Model | Baseline avg | V-GIFT | Delta |
| LLaVA-1.5-Vicuna-7B | 53.6 | 54.5 | +0.9 |
| LLaVA-1.5-Qwen2.5-7B | 57.8 | 58.7 | +0.9 |
| LLaVA-OneVision-1.5 | 65.7 | 66.9 | +1.2 |
LoRA widens the gap: on the Qwen backbone the average moves from 57.4 to 59.6 (+2.2), with CVB-2D +3.9 and BLINK +3.3, above VIRAL, which adds an auxiliary distillation loss in the same LoRA setting. On OneVision, BLINK goes from 48.8 to 52.2 (+3.4).
A compute-matched control on OneVision retrains on 3% extra ordinary instructions and the average falls from 65.7 to 65.6. The same 3% as SSL tasks reaches 66.9. Putting SSL before instruction tuning does almost nothing (57.7 vs 57.8). Putting it after causes catastrophic forgetting (average 39.9). Each pretext alone adds about +0.4 on average; all three together reach +0.9. Even rotation samples cropped from a single high-resolution image lift the Qwen average from 57.8 to 58.4.
TVI, a score for how much the answer depends on visual tokens, rises from 0.1238 to 0.1368 on Vicuna CVB-2D, and barely moves on MMStar (0.1426 to 0.1430). Attention maps sit more tightly on lamps and TVs. Generalist suites do not collapse: MathVista on Qwen goes 12.2 to 15.3, RealWorldQA 56.4 to 59.0. Vicuna and OneVision stay roughly flat; OneVision MathVista dips from 22.9 to 22.6.
This is a cheap data-side knob. No new vision encoder, no reconstruction loss, no RLVR. Mix in tasks that cannot be solved without the image, and the LLM decoder starts using visual features it already had. For teams still training LLaVA-style models, that is easier to ship than another projector or a DINOv2 distillation head.
The lift is incremental. About one point on average, and Vicuna full-SFT BLINK even drops from 38.2 to 37.8. Groups already running jigsaw-style RLVR for visual post-training should read this as a lighter alternative, not as a replacement result.
There is no dedicated Limitations section. The numbers still show the cracks.
Gains are small and uneven. Vicuna POPE barely moves (87.0 to 87.2). OneVision peaks at ρ=3%; 10% and 30% do not keep climbing. ρ has to be retuned per model.
The comparisons are incomplete. VIRAL is LoRA-only and omits BLINK. There is no matched bake-off against recent jigsaw-as-verifiable-reward pipelines. TVI is reported only for Vicuna on two suites.
The pretext tasks are low-level. Rotation, color matching, and correspondence force the model to look, but they sit far from chart reading or scientific reasoning. Only projection-style LLaVA architectures are tested. Whether cross-attention MLLMs eat the same mix is unknown.