Reframe: Observation-Aligned Multi-Teacher On-Policy Distillation for Agentic Visual Reasoning
Haozhi Yuan, Handong Li, Yifan Zhang, Qianshan Wei
cs.CV
2026-11-10
Reframe post-trains Qwen3.5-9B to use tool-returned views, hitting 69.48 image and 53.10 video averages, with +5.23 tool gain.
Visual agents crop images and pick frames, then often answer as if the returned view were never there. That failure is observation neglect: the tool fired, and the next prediction barely moves.
Multi-teacher on-policy distillation (MOPD) hits a related wall. The student samples its own trajectory, and a specialist teacher scores every token. The teacher is stuck with whatever view the tool just returned. A bad crop or a bad frame pick cannot be undone by a token-level loss.
Mixing image and video in one run tends to leave the shared model behind either specialist. Off-policy supervised fine-tuning tangles two tool habits, and RL then assigns one reward to the whole trajectory. Reframe splits the job into two stages and skips a task-specific SFT cold start.
The base model is Qwen3.5-9B. The only tools are CropImage and SelectFrames. Reframe-100K holds 63K image questions and 37K video questions. Each box is scaled by 1.5 around its center, and each temporal span grows by one frame on both ends. Questions answerable from text alone are dropped. Full inputs and evidence-only inputs are each rolled out 8 times; items that are always solved, never solved, or unsupported by their evidence are removed. Both stages use this pool.
GD-GRPO trains a spatial specialist and a temporal specialist separately. The reward covers the answer, the decision to call a tool, overlap with the annotated evidence, and format validity. Overlap is an IoU. On images, crops along one trajectory are matched one-to-one, so hitting the same object twice does not stack the reward. On video, only the latest frame selection counts, and later attempts are discounted. A well-localized call is paid even when the answer is wrong. The extra call bonus fades as training proceeds.
A correct crop can still be ignored, because the original picture is a shortcut. After the first successful call, training drops that original view with probability 0.5 and keeps the question plus the tool return. At inference the original view stays. Advantages are normalized inside subgroups that share a pre-sampled drop flag, so a harder dropped prompt is not scored against an easier kept one. Group assignment does not depend on whether the policy actually called the tool.
OG-MOPD then folds both specialists into one student. If a tool action is missing, invalid, or below an IoU threshold against the annotation, it is swapped for a canonical action taken from that annotation. The environment runs the repaired action, and the student continues from the new view. Tokens the student sampled still get MOPD from the matching specialist. Injected action tokens were not drawn from the student, so they take a separate negative-log-likelihood term. Inference uses neither the gate nor the teachers.
| Method | Image avg | Video avg |
| Qwen3.5-9B | 63.46 | 49.75 |
| Image-RL | 67.92 | 50.39 |
| Video-RL | 66.35 | 52.78 |
| Mix-RL | 66.75 | 51.41 |
| Vanilla MOPD | 68.19 | 52.32 |
| Reframe | 69.48 | 53.10 |
Image-RL and Video-RL are the single-modality GD-GRPO specialists. Mix-RL uses the same reward on mixed data and still trails them by 1.17 and 1.37, keeping 74% and 55% of their gains over the base. Vanilla MOPD keeps 106% and 85%, and beats the matching specialist on 5 of 14 benchmarks. Observation repair adds 1.29 on images and 0.78 on video. The largest step versus MOPD is VisualProbe, 52.62 to 57.48. Against the base, VisualProbe is +13.60, ZoomBench +9.35, and V +7.85.
Outcome-only GRPO, with no grounding reward and no input drop, scores 65.85 on the image average and gains +1.97 when tools are on. GD-GRPO scores 67.92 and +3.82. Tool-free accuracy rises by only 0.22. On V, observation dependence moves from 52.59 to 69.03: the share of errors under the average of random and blank crops that flip once the ground-truth crop is supplied. GD-GRPO is higher by 5.23, 3.67, and 3.14 under GT, random, and blank returns. The largest piece is the true crop.
On the external table Reframe sits at 69.48 and 53.10, 11.71 and 8.86 above the strongest other system listed. The base already accounts for much of that gap. Untouched Qwen3.5-9B is at 63.46 and 49.75, ahead of the final averages of Pixel-Reasoner, Thyme, DeepEyes, and DeepEyesV2. Within-model diagnostics travel better. Final dependence on V is 75.53, against at most 41.93. With the original image still present, a ground-truth crop scores 96.86, a random crop 85.86, and a blank crop 88.48. Tool gain over seven image benchmarks is +5.23, against at most +1.47. DeepEyesV2 drops 2.45 when tools are enabled. About 87% of Reframe's gain over the tool-free base comes from tools; the no-tool score is 64.25.
Cross-modal transfer is uneven. Video-RL lifts the image average to 66.35. Image-RL lifts the video average only to 50.39.
Code, the 100K set, and the model are released. For a 9B policy whose tools are crop and frame select, mixed RL gives back part of the specialist gain. Teaching each specialist to read the returned view, then merging them with dense token feedback, puts the shared policy slightly above both specialists on average.
What moves is tool use. Tool-free accuracy barely changes. The gate is about one point over vanilla MOPD, and it needs box or segment labels to build the repaired action. Without that annotation, stage two does not transfer as written.
The IoU threshold and the reward coefficients are absent from the main text. Dependence is measured only on V, and tool gain is averaged only over seven image benchmarks. Video has no matching diagnostic. Decoding is greedy.
The lead over four published visual agents is largely the base model. A shared visual-token budget is not a shared starting point. The introduction argues that SFT-then-RL leaves a unified model behind, but that pipeline is not in the tables. The controls are Mix-RL and vanilla MOPD.
An average win over the specialists is not a win on every set. Image-RL still leads HR-Bench-4K, 85.25 to 84.75, and PerceptionBench, 36.23 to 36.07. The video average is only 0.32 above Video-RL, and Video-RL remains ahead on MVBench and MLVU. Injecting a canonical action rewrites the history the student continues from, so that span is off-policy. How often the student later emits those repaired actions on its own is not reported.