Select, Compress, Reinvest: A Controlled Study of Visual-Token Allocation in Long-Video MLLMs
Prakhar Khatri
cs.CV, cs.CL
2026-09-03
Controlled select/compress/reinvest: on hour-long videos, eight OMP frames beat sixteen uniform ones by 6.9 points; halving resolution costs at most 0.44, reinvestment adds 2-3.
A video language model cannot see every frame. An hour decoded at 1 fps is 3,600 images; typical systems keep eight, thirty-two, or sixty-four. Which frames survive is usually treated as preprocessing. This paper asks whether that is the right attitude, by holding the scorer, the answerer, and the eval harness fixed and changing one allocation decision at a time: which timestamps, how many pixels per frame, and what the recovered tokens buy.
Published selectors make that comparison hard. AKS, FOCUS, LDDR, and Q-Frame typically move the scorer, the resolution policy, the answerer, and the frame count together. A number from paper A versus a number from paper B compares two experiments, not two ideas.
Every contrast is paired: the same question answered twice, one input policy changed, McNemar on correctness. Candidates are decoded at 1 fps and encoded once with LongCLIP on the question stem, never the answer options. All six selectors read that cache.
The rules are uniform sampling, cosine top-k, AKS, FOCUS⋆ (a replay of the published clip-bandit schedule on dense LongCLIP scores, not the original budgeted ITM scorer), Orthogonal Matching Pursuit from 1993, and LDDR's stage-1 Linear-DPP. OMP repeatedly takes the frame most correlated with the current residual, then projects that span out of the query so later picks cannot repeat it. No video-specific machinery, no hyperparameters to tune. Any gap against a purpose-built rule is then a gap in the rule, not in tuning effort.
Compression freezes the chosen timestamps and cuts the per-frame spatial budget by about half (D@53). Reinvestment spends the savings on sixteen compressed frames against eight full-resolution ones. On LongVideoBench the measured token ratios are 0.996 and 0.984, so the winning arm is also the cheaper one. The main answerer is Qwen3-VL-8B with subtitles off; InternVL3 2B/8B and GPT-5-mini check transfer. Benchmarks: full LongVideoBench (n=1337, stratified at 15/60/600/3600 s), Video-MME (n=2700), LVBench (n=1549).
Selection is the large lever. On hour-long LongVideoBench, OMP at eight frames scores 0.5461 against 0.4770 for uniform sampling at sixteen: 6.9 points on half the input (p=0.0011). The ten-minute bin repeats the pattern: OMP-16 at 0.6578 versus uniform-32 at 0.6044, +5.3. On 15-second clips, uniform, top-k, and OMP are identical at 0.7249. Once the pool fits in the budget, the selector has nothing to do.
Matched comparison at k=8 with Qwen3-VL-8B:
| Method | LongVideoBench | Video-MME | LVBench |
| Uniform | 0.5654 | 0.5637 | 0.3454 |
| top-k | 0.6028 | 0.5704 | 0.4319 |
| OMP | 0.6223 | 0.6222 | 0.4635 |
| AKS | 0.5916 | 0.6059 | 0.4287 |
| FOCUS⋆ | 0.5819 | 0.5578 | 0.3983 |
| LDDR-select | 0.6320 | 0.6193 | 0.4693 |
OMP beats uniform by 5.69 / 5.85 / 11.81 points. LDDR-select is the only rule that stays within a point on all three. Swapping LongCLIP for SigLIP replaces 67–84% of OMP's frames on the 600 s bin; the order uniform < top-k < OMP still holds.
Compression is close to free. Across three Qwen aggregates, freezing timestamps and halving the spatial budget moves accuracy by at most 0.44 points. On the pooled LongVideoBench long bins the 90% interval sits inside a post-hoc ±3-point equivalence margin and misses ±2. Spending the savings on sixteen frames then adds 2.24 on LongVideoBench, 3.04 on LVBench (p=0.0009), and 1.56 on Video-MME. Uniform sampling points the same way (+1.13); the selector-by-budget interaction is not significant, so this does not have to be an OMP-specific effect.
An implementation bug is part of the result. A padding branch in the AKS port collapsed to global top-k for every k<32, changing about 99.5% of selected frames. After the fix, LVBench moved +0.07, LongVideoBench −1.05, Video-MME +2.74. Against Chen et al. 2026a, same answerer family, budget, and encoder, eight of nine comparable cells are lower here, by 0.07 to 3.74 points, and the uniform floor on LVBench is 28.08 versus 34.54.
The engineering order is clear. Get the timestamps right first, treat resolution as slack, and spend the recovered tokens on more moments rather than pocketing them. An unmodified 1993 pursuit algorithm matches purpose-built selectors, so a new rule should at least include a classical greedy or pursuit baseline on the same scores.
The sharper claim is about evaluation. The gap between two controlled harnesses already covers most of the deltas that published selector papers advertise. Comparing "two points higher" across papers is not informative. Scoring on the stem versus stem-plus-options moves another two to four points, and a query built from the wrong options captures about half of that gain, so the axis is not even clearly about relevance.
FOCUS⋆ and LDDR-select are controlled reproductions, not full-system ports. The ±3-point equivalence margin was chosen after seeing the interval, so it is a description of this sample, not a preregistered equivalence claim. Residual geometry is LongCLIP-specific; the failure audit is purposive and cannot estimate rates. Selection still needs 1 fps decode plus a full-pool LongCLIP pass, a cost that is not in the answerer token budget and does not amortize for a single query per video. InternVL3-8B is almost insensitive to OMP on medium and long Video-MME (+0.33, +0.11) while gaining 10.14 on LVBench: selection effects belong to an answerer–benchmark pair, not to a timestamp set. Subtitles are off throughout, which hurts every visual selector on subtitle-anchored items. MDP3 and Q-Frame are absent.