Region-Level Policy Optimization for Fine-grained MLLM Perception
Yuheng Shi, Xiaohuan Pei, Minjing Dong, Chang Xu
cs.CV
2026-09-17
Region-level RL on a proposal head lifts Qwen3.5-9B to 80.1 avg; at 576 tokens the 4B system beats the base model at 4096.
When multimodal LLMs miss fine detail, the default fix is higher input resolution. Visual tokens grow, and both the vision encoder and the language-model prefill get more expensive. Fine-grained VQA is two operations: find the region of interest, then read what is in it. On ZoomBench with Qwen3.5-4B, a 238-case diagnostic (full-resolution correct, question-only incorrect) shows localization tolerates about 3 to 4× stronger token compression than recognition. The box still re-predicts at 25× compression; reading the crop fails at 9×.
That gap argues for localizing on a coarse view and spending resolution on the chosen evidence. Decoding coordinates with the MLLM is a full extra pass and leans on grounding skill. Distilling response-to-image attention into a lightweight proposal net (SD-RPN) is fast, but it copies attention noise, and the map reaches the answer only after binarization and connected components, so answer gradients never hit the map.
Vision-RL2 updates only the proposal network. The MLLM is a frozen reader. The RoI heatmap is split into connected regions, each treated as an action. The reader teacher-forces the gold answer and scores a region by how much the answer log-odds drop when that region is removed. No response sampling and no reasoning trace: n regions cost n+1 reader forwards.
Two policies run together. The subtractive group does leave-one-out on the current proposal: if a region's contribution sits below a noise margin estimated from control patches, its confidence is driven down; above the margin, it is kept. The additive group pulls residual regions from multi-layer response attention that the proposal missed, and lifts those with positive contribution. Region extraction is not differentiable; credit flows through mean region logits or mean log-probabilities back into the dense map. A KL anchor to the initial SD-RPN, plus an attainability weight on gold likelihood, damps samples the reader cannot answer under any mask.
At inference the proposal net emits an RoI from the image and question, with no decode. Sparse visual encoding zooms the crop by foreground occupancy and keeps only foreground tokens. Position embeddings are assigned on the full box grid before dropping background, so geometry is unchanged. Source-image tokens are not re-encoded; part of the proposal-pass KV cache is reused.
Backbones: Qwen3.5 4B/9B, Qwen2.5-VL-7B, and encoder-free Gemma-4-12B. RL data is 7K filtered VisualCoT items, one epoch, 576 visual tokens on the source image during training. The main table gives every method a 16,384 source-token cap.
| Model | V Bench | ZoomBench | Six-benchmark avg |
| Qwen3.5-9B | 83.8 | 54.9 | 74.6 |
| Vision-OPD-9B | 90.6 | 65.1 | 78.7 |
| Gemini-3.1-Pro | 88.0 | 61.2 | 79.3 |
| Vision-RL2-9B | 95.3 | 68.4 | 80.1 |
| Vision-RL2-4B | 91.1 | 65.1 | 75.3 |
| Vision-OPD-4B | 90.6 | 59.5 | 76.0 |
The 9B model averages 80.1, above Gemini-3.1-Pro and Vision-OPD-9B in this table, with the best V and ZoomBench scores overall. The 4B model beats Vision-OPD-4B on those two and on HR-Bench, and trails on MME-RealWorld, where full finetuning can change the reader, a knob this frozen design gives up. Qwen2.5-VL-7B averages 71.0 versus ZwZ-7B at 69.9. Gemma-4-12B averages 72.8 versus 69.0 for SD-RPN and 63.3 for the base model.
Under the training-aligned short-answer protocol, the 4B system at a 576-token budget averages 71.1, +4.5 over SD-RPN (66.6) and +14.9 over the base model (56.2), and it beats the base model at 4096 tokens with about one quarter of the tokens. Region-level RL alone is +3.0; sparse encoding adds +1.5. The RPN head rides the answer prefill and routes in 31 to 49 ms; attention routing or coordinate decoding takes 1.1 to 1.8 s.
Swapping the functional score for generation accuracy costs 2.3 points, raw log-probability costs 1.5, cell-level keep-sets cost 1.2, dropping the noise margin costs 1.1, and dropping the additive group costs 0.9.
Fine-grained perception does not require full-model RL or think-with-images loops. A reusable, decode-free localizer, rewarded by a frozen reader's functional score, can spend resolution on evidence. Only the proposal head is trained, an order of magnitude fewer parameters than full finetuning, and the same recipe covers four backbones.
The 4B gap versus Vision-OPD on MME-RealWorld is the honest trade: this method improves routing, not the reader. The 80.1 main-table average and the 4.2× token claim come from two different protocols. Do not mix them.
There is no dedicated limitations section. Training needs gold answers, so open-ended settings do not plug in. The frozen reader cannot close MME-RealWorld-style gaps that need recognition itself to change; full finetuning still wins there. Additive candidates come from response attention, so training still runs a generation-side attention pass. The policy starts from SD-RPN, so cold-start quality is bounded by attention pseudo-labels. Multiple regions share one enclosing box; the paper does not isolate how much sparse encoding saves on fragmented evidence. The main table uses an LLM judge; ablations use rule-based short answers. Those numbers are not interchangeable.