Orthogonal projection keeps VideoLLM SFT at 89% while cutting the text shortcut from +30 to +6

Suppressing the Shortcut Hidden Behind the Gains of VideoLLM Fine-Tuning

Youngwoo Shin, Jongsuk Kim, Junmo Kim

cs.CV

2026-11-19

VETO projects LoRA off the text-only subspace. Seven VideoLLMs match SFT (89.2% vs 89.6%) as the text shortcut drops from +30.5 to +6.0, beating SFT on six shifted benchmarks.

What problem this solves

Supervised fine-tuning is still the default step when a VideoLLM is adapted to video question answering. Work from KAIST and PYLER measures how much of the accuracy gain ignores the video. Something-Something v2 (SSv2) turns action templates into multiple choice. Options in one family share the objects and differ only in what happens over time, such as a knife moving up or down. A blind filter drops any item the base model answers from text alone on three tries with the options shuffled. After that filter, Qwen2.5-VL-7B scores 49.1% with the video and 19.3% from text, under the 25.7% chance rate. Rank-16 LoRA SFT then reaches 89.6% with the video and 49.8% from text. The extra 30 points are a pathway fine-tuning opened: option wording points at the answer.

The pathway is already above chance at epoch one, obvious at 1,000 items, and still growing near 57,000. Across seven backbones, five language-model families, and sizes up to 32B, SFT lifts text-only accuracy by at least 20 points over each base model. The filter does not remove regularities left in the option sentences.

Inside the training distribution, full-input accuracy rises with the shortcut, so the gain hides it. Perception Test uses different videos, questions, and option wording. The same SFT checkpoint stays at the base model's 61.1% with the video, while text-only accuracy rises another 17.0 points. Where the regularity fails, the shortcut still answers, and the gain from the video is gone.

Method

VETO, Video-Evidence Training via Orthogonal projection, blocks the directions in which that pathway would be written. It does not name the regularity.

Calibration runs once per backbone. Four hundred items are drawn from the same pool, stratified across 16 action families, held out of training, and used without labels. The base model sees only the question and options. Input activations of every linear module form an uncentered second-moment matrix, so the mean direction stays in the subspace. An eigendecomposition keeps the smallest set of leading directions whose eigenvalues cover at least a fraction ρ of the trace. The default is 0.95. On Qwen2.5-VL-7B the input width is 3,584 and the average rank is about 250: roughly 7% of the dimensions cover about 95% of text-only energy. A different draw spans nearly the same subspace.

Training matches SFT: LoRA rank 16, 16 frames, learning rate 1e-4, at most 10 epochs. After every optimizer step the input-side factor is right-multiplied by the projector. The component inside the subspace is zeroed, and the orthogonal complement is kept. On those directions the response stays that of the base model. The language prior already there remains. The new text-only pathway cannot be written.

Video still leaves writable directions. With no video, 95% of text-position energy lies in the subspace. With the original video that share falls to about 0.88. The update is written on the part that moved out.

Timing decides whether the constraint prevents the shortcut or only erases it later. Projected once at initialization, the shortcut returns to the SFT level: text gain +30.0, full accuracy 88.7%. Projected once after training, the text gain falls to +2.4 and full accuracy falls to 82.4%, and what was learned along those directions is cleared. Projected every step, full accuracy stays at 89.2% and the text gain is +6.0.

Results

Shift benchmarks are blind-filtered the same way. The answer is the option letter with the highest logit. Means below are Qwen2.5-VL-7B on SSv2, three seeds.

MethodFull inputText onlyText gain
Base49.1%19.3%0
SFT89.6%49.8%+30.5
VETO89.2%25.3%+6.0
Random subspace, same rank89.6%49.6%+30.4
Options reshuffled each epoch89.4%49.4%+30.2
LMH83.4%32.6%+13.3
mDPO87.0%45.3%+26.1

VETO's text-only score, 25.3%, sits on the 25.7% chance line. A random subspace of the same rank barely moves the shortcut, so the removed rank is not what matters. At ρ = 0.99 the text gain is +1.1 and full accuracy is 89.5%. At ρ = 0.90 the text gain is +12.9 and full accuracy is 88.6%.

On Perception Test, SFT matches the base model at 61.1% and adds +17.0 text-only points. VETO reaches 64.3% with a text gain of +4.9. Physics is 63.0% against SFT's 59.0%. Semantics is 78.5% against 75.0%, and SFT sits under the base model's 78.0%.

BenchmarkBaseSFTVETO
Vinoground55.9%63.5% (+13.5)66.9% (+7.7)
AoTBench41.3%53.1% (+10.9)55.2% (+5.6)
MVBench58.0%56.5% (+7.8)58.0% (+3.5)
TVBench42.4%44.3% (+7.6)46.4% (+3.0)
TempCompass64.0%65.8% (+4.0)67.6% (+2.4)

Parentheses are text-only gains over the base model. On MVBench, SFT falls to 56.5%, below the base model's 58.0%, while its shortcut still grows.

Figure 5 reports shortcut reduction, 1 − ΔVETO/ΔSFT: 71% at 3B, 80% at 7B, 88% at 32B, 95% for Molmo2-O-7B, 64% for Qwen3-VL-8B, 78% for LLaVA-OneVision-7B, and 76% for Gemma4-E2B. Full-input gains stay close to SFT.

Entropy over the options, as a share of its maximum, shows how settled the answer is. On SSv2, SFT is at 16% without video and 2% with it. VETO is at 68% and 2%. The share the video removes is 14% under SFT and 66% under VETO. On Perception Test the gap is 19% versus 63%.

One epoch costs 4.40 GPU-hours for SFT and 4.67 for VETO, about 1.06 times, plus 0.03 GPU-hours of calibration. LMH costs about 5.05 times SFT, mDPO about 3.18 times.

On NExT-QA, full-input accuracy is 70.9% for the base model, 71.7% for SFT, and 71.6% for VETO, with text gains of +8.1 and +1.3. On STAR the scores are 67.9%, 70.5%, and 70.9%, with text gains of +11.6 and +1.3. The shortcut is suppressed again. On both sets the SFT gain on the full input is small to begin with.

Why it matters

An SSv2 score of 89.6% does not show that the model reads the video. The same SFT weights add nothing over the base model on Perception Test. What rose is the text pathway. VETO changes neither the data, the loss, nor the schedule. Each step adds a projection. The in-distribution gain mostly stays, and all six unseen benchmarks come out ahead of SFT.

The architecture is unchanged. The fit is narrow: an existing VideoLLM, multiple-choice targets, LoRA, and a worry that the model memorizes option sentences. The main knob is ρ. If the task actually needs a stable link between wording and the answer, new uses of text inside the subspace are blocked with the shortcut. The base model's existing language prior stays. What is removed is the pathway fine-tuning would have added.

Limitations

The shortcut is defined by text-only accuracy, which multiple choice supplies and free-form answers do not. The projection is implemented on the input side of LoRA. Full fine-tuning and other adapters are untested. The main runs train on SSv2 only. NExT-QA and STAR report in-distribution accuracy, not the six shift benchmarks.

The calibrated subspace is incomplete. On the six shift sets, about one fifth of text-only energy falls outside it, and the text gain does not go to zero. Perception Test still shows +4.9. Inside the subspace every new use of text is blocked, valid reasoning included. Where the displaced learning goes is not measured.

The scores are not comparable to published full-benchmark numbers. The blind filter keeps items the base model cannot yet solve from text. LMH and mDPO use one seed each. Gemma4-E2B's base model scores 25.8% with video and 25.7% from text, essentially chance, so a large reduction ratio there is not evidence that video reading was restored. The outputs are not shown to be faithful to the frames either. Biases in the base model remain.

Terms

Source

Related papers

All paper explainers