EviRover: Reinforcing Agentic Perception Beyond a Glance
Kaixuan Fan, Kaituo Feng, Tianshuo Peng, Yilei Jiang, Manyuan Zhang, Junke Wang, Xiangyu Yue
cs.CV, cs.AI
2026-10-01
EviRover reframes grounding, segmentation, and counting as evidence seeking: a 4B model trained with SFT plus agentic RL decides whether to zoom in, search the web, or answer directly, beating its backbone by 30.2 points on the new EviLens benchmark and matching proprietary systems.
Perception tasks have long carried a default setup: given an image and a query, the model outputs a box, a mask, or a count in a single forward pass. The setup assumes the image plus the model's parametric knowledge always suffice.
Often they don't. A target occupying under 0.1% of a high-resolution image is invisible at a glance. The person who served as dean of some institute from 2003 to 2008 cannot be identified without a search. The paper calls these cases perception under insufficient evidence, and notes that the model still has to commit to a location or a boundary with no second chance. Existing multimodal agents do interact, but their interaction serves a textual answer. A box drawn on the image cannot be produced by copying a search result, which makes perception a stricter test of evidence seeking than question answering.
EviRover builds on Qwen3-VL-4B-Instruct and trains in two stages, with the model itself deciding which kind of evidence a query lacks and which action can supply it.
Data first. Two pipelines match the two forms of missing evidence:
Query construction has one crucial step. A query that names the target directly needs no search, so each query is rewritten through iterative entity replacement: swap a named entity for an indirect description, test whether Seed-1.8 can still resolve the query without tools, and stop when it cannot, up to 20 hops.
SFT trajectories come from Gemini-3.0-Pro and Seed-2.0-Pro, keeping only those with logically consistent action sequences and correct answers. One trap deserves mention: when the annotation model already knows the person, it answers in one step, and a 4B backbone imitating that trajectory learns to answer without evidence, which is hallucination training. Those trajectories are downsampled, with a portion kept so the model still answers directly when its own knowledge suffices.
Then training. SFT establishes the interaction format and basic tool use; RL uses the correctness of the final perceptual output as reward. The toolkit has ten tools: four for external retrieval (text search, text-to-image search, reverse image search, browsing), three for fine-grained inspection (crop, drawing candidate boxes back on the image, cropping a candidate region for final confirmation), and three for task-specific prediction (SAM3 mask generation, side-by-side panel comparison, a Python interpreter).
RL rewards are task-specific: box IoU for grounding, mask IoU for segmentation, exact match for counting, and a soft set-level F1 for spot-the-difference. The spot-the-difference reward decouples threshold from weighting: the matching threshold sits at 0.1, well below the evaluation threshold of 0.5, so roughly localized boxes earn partial credit and the learning signal stays dense; each match is weighted by IoU raised to α = 2, which crushes the value of weak matches and prevents the model from farming reward with extra coarse boxes. For segmentation, when SAM3 fails to return a mask the trajectory is not given zero reward, since under group-relative advantage estimation a zero shifts the advantages of every other rollout in the group; the predicted box is used as a rectangular mask instead.
Reward distributions differ a lot across tasks, so EMA-GRPO maintains per-task running statistics for normalization. The policy-loss reduction switches from token-mean to sequence-mean-token-mean, with a full derivation in the appendix: token-mean amounts to length-biased sampling over trajectories, and in agentic RL trajectory length is itself policy-dependent, since retries and redundant interactions all add tokens that have no claim to a proportionally larger update.
EviLens is the accompanying human-verified benchmark: 688 instances across localization (140), recognition (182), spot-the-difference (15 images, 79 annotated differences), segmentation (195), and counting (156). Agentic methods get at most 35 tool calls per trajectory.
| Method | Task | Metric | Result |
| Gemini-3.5-Flash | Localization | IoU | 0.373 |
| EviRover (4B) | Localization | IoU | 0.444 |
| Seed-2.1-Turbo | Segmentation | gIoU | 0.779 (best overall) |
| EviRover | Segmentation | gIoU | 0.686 |
| EviRover vs backbone | Counting | Acc | 50.0 vs 19.2 |
| EviRover vs backbone | Spot-diff | micro-F1 | 0.171 vs 0.005 |
| GPT-5.6-Sol | Spot-diff | micro-F1 | 0.112 |
Localization is where EviRover beats the proprietary field most clearly. On counting it lands 1.3 points behind GPT-5.6-Sol. GPT-5.6-Sol's spot-the-difference score is oddly low, and the paper's explanation is informative: it keeps predicting a single box even when the prompt says there are several. Open-source grounding specialists score near-zero IoU on localization and segmentation specialists top out at 0.343 gIoU, which supports the paper's claim that the bottleneck is evidence acquisition, not perceptual precision.
Averaged over the five primary metrics, EviRover beats its backbone by 30.2 points. The gains transfer: on WebEyes it reaches 40.0 IoU and 48.5 [email protected] on grounding, ahead of every evaluated model including the proprietary ones, and beats Pixel-Searcher-8B by 10.4 gIoU on segmentation. On ReasonSeg and RefCOCOg it improves over the backbone on every split and metric. Among general multimodal benchmarks, MMMU through MathVerse gain 3 to 4 points, and BrowseComp-VL more than doubles from 9.5 to 24.8.
The contribution splits into three layers.
Conceptually, the framing of perception under insufficient evidence pulls a scattered set of failures, tiny targets, knowledge-dependent grounding, cross-panel comparison, into one setting, and observes that perceptual outputs are harder to fake than text answers since a box can be checked directly against spatial annotations. This framing will likely be cited again.
For practitioners, several pipeline designs are directly reusable: multi-hop entity replacement with termination once a strong model can no longer answer without tools; downsampling annotation-model trajectories that resolve in one step, a hallucination source that any agent-trajectory distillation effort should watch for; and the decoupled threshold-and-weighting reward pattern.
On results, a 4B model approaching proprietary flagships on a benchmark the authors built themselves deserves a discount, since EviLens comes from the same distribution as the training data. The transfer numbers on WebEyes and conventional benchmarks carry more weight. The 15.3-point absolute gain on BrowseComp-VL is the strongest number in the paper, because that task sits farthest from the perception training distribution.
Stated by the authors: only Gemini-3.0-Pro could complete the localization annotation at the time, tying annotation quality to a single proprietary model; real-person synthesis breaks beyond roughly three references, capping identity coverage; and the spot-the-difference subset holds just 15 images with 79 differences, with micro-F1 chosen as the primary metric precisely because the subset is small.
A few things hold up less well on inspection. Category sizes differ by an order of magnitude, so the 30-point average is dominated by the larger categories. Counting queries require every individual in a photo to be identity-verified, which narrows the image distribution and makes the 50.0 accuracy hard to extrapolate. A 35-call tool budget means inference costs orders of magnitude more than a single forward pass, and the paper says nothing about latency or call counts. Finally, SFT trajectories are generated by Gemini and Seed models that reappear as closed-source baselines, so the training data already contains capabilities from the systems being compared against, a point the paper leaves undiscussed.