Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMs
Yunheng Li, Guohong Mu, Hao Li, Shengsheng Qian, Dingwen Zhang, Qibin Hou, Ming-Ming Cheng
cs.CV
2026-08-21
OraRL turns annotations into oracle rollouts with a decoupled advantage baseline. Video-ORA-9B hits 66.0 temporal mIoU and 73.1 on VSI-Bench, versus 55.0 for GPT-5.
Fine-grained video perception is a post-training problem, not a scale problem. On temporal grounding, tracking, segmentation, and spatial intelligence, GPT-5 and Gemini-3-Pro often lose to open specialists under 10B. SFT treats each annotation as a maximum-likelihood target: the model copies the output format and never learns to tell a near miss from a miss. Group-relative methods such as GRPO score several on-policy rollouts against the annotation, but the annotation is only a scorer. Precise intervals, boxes, masks, and trajectories almost never appear in the sampled group, so many groups have no reliable positive. Chain-of-thought makes every rollout longer and more expensive, without lifting accuracy.
Adding prompts does not close the gap. Sample efficiency is the bottleneck.
OraRL's rule is annotation-as-rollout: serialize the annotation into the model's response format and append it as an oracle rollout. Nothing on-policy is dropped. Multiple-choice answers, time spans, boxes, box trajectories, and timestamped segmentation prompts all fit the same interface.
Mixing the oracle into the group naively fails. A high-reward oracle lifts the group mean, so rollouts that beat the on-policy mean can still get a negative advantage. That failure is advantage inversion; the inverted band grows with the oracle-policy reward gap. Naive injection drops a three-task average from GRPO's 60.3 to 55.4, and tracking by 11.9 points. Across 11,503 groups and 92,024 rollouts, 22.4% of GRPO-positive samples flip sign; 42.5% of groups see at least one flip, and 8.3% lose every positive. Tracking is worst, at a 38.7% flip rate.
OraRL splits the estimator:
Tempsamp-R1 fought inversion with hand-designed reward shaping per task. OraRL computes advantages from on-policy rewards and the oracle-policy gap, so one update covers any annotation that can be written as a response.
Video-ORA is trained on Qwen3.5 with about 285k SFT prompts and 100k RL prompts, one recipe across seven task families, and no CoT at decode.
| Task | Metric | Video-ORA-9B | Baseline |
| TimeLens temporal grounding (3 sets) | mIoU macro | 66.0 | TimeLens2-8B 62.5 |
| GOT-10k tracking | AO | 78.2 | OneThinker-8B 73.0 |
| MeViS video segmentation | J&F | 61.3 | OneThinker-8B 52.7 |
| VSI-Bench | macro | 73.1 | GPT-5 55.0, Gemini-3-Pro 55.1 |
| Spatial trio | VSI+MMSI+MindCube | 56.1 | prior best 51.0 |
Per-split mIoU is 61.8 / 63.6 / 72.5 against TimeLens2-8B's 58.6 / 58.6 / 70.2. The [email protected] gap on ActivityNet is 6.7 points: coarse retrieval is already in the backbone, exact boundaries are what sampling rarely hits and the annotation always supplies. Video QA macro rises from 61.9 on the backbone to 66.8, with +15.2 on VideoHolmes. All eight RefCOCO-family [email protected] splits rank first. On STVG, spatial [email protected] jumps 15.5 over Qwen3.5-9B while temporal [email protected] moves only 3.0. Segmentation is the largest absolute change versus the backbone: MeViS J&F 32.1 to 61.3, ReasonVOS 21.5 to 63.7, because the backbone cannot emit usable mask prompts.
From 0.8B to 9B, every scale beats its Qwen3.5 twin; the macro average climbs from 51.8 to 66.2. From 6.4k to 100k prompts, OraRL gains 5.2 on a video-perception aggregate and 3.6 on spatial intelligence, against 2.8 and 3.1 for GRPO. CoT does not help: GRPO with CoT averages 58.5 on temporal grounding versus 58.7 without, while step time rises from 93.9 s to 135.6 s. OraRL, answer-only, averages 61.4 at 62.4 s/step. Relative to SFT that is about 2.2×, against about 4.9× for GRPO with CoT. Pruning at κ=0.5 cuts step time from 92.5 s to 62.4 s for a 0.4-point drop, and peak memory from 62.4 GB to 50.9 GB. On ten-minute videos at 2 fps (120k visual tokens), median end-to-end latency is 24.3 s versus 29.0 s for the CoT backbone; generation after first token is 130 ms versus 4,780 ms.
Positive-sample scarcity in video RL is structural. The annotation is the most reliable positive in the group, not only a reward function, and it has to sit outside the baseline or naive mixing loses to GRPO. For anyone training a unified video MLLM, this is a CoT-free recipe that still scales from 0.8B to 9B and keeps gaining out to 100k prompts. Code, data, and weights are public.
The gain is incremental: the objective is still GRPO's clipped importance ratio. The change is group composition, advantage estimation, and pruning.
Every annotation must serialize into a valid oracle and a scalar task reward. Ambiguous, partial, or noisy labels, and learned oracles, were not tested. Route planning still trails Gemini-3-Pro and GPT-5. The MMSI-Bench plus MindCube mean is 47.7, below Grok-4 at 50.7. On the corrected ReVSI protocol, Video-ORA-9B drops 14.9 points from VSI-Bench while the backbone drops 6.3; at 128 frames, relative-direction accuracy is 91.5% forward and 8.3% backward, which matches the forward-only templates in the VSI-590K training mix.
Most ablations use a three-task protocol; the full seven-task mix is reported only for the final model. A LUFFY-style teacher-trace injection averages 54.7 and tracking falls to 50.0, so stuffing a better sample into the group is interface-sensitive. The decoupled estimator is load-bearing.