SpatialCORE turns BBox coordinate entropy into an RL reward, lifting Qwen3-VL-8B spatial reasoning

SpatialCORE: Confidence-Aware Grounded Spatial Reasoning in Large Vision--Language Models

Rafi Ibn Sultan, Xiangyu Zhou, Md. Sajid Alam Chowdhury, Chengyin Li, Prashant Khanduri, Marco Brocanelli, Dongxiao Zhu

cs.CV, cs.AI

2026-09-30

GRPO post-training that rewards confident BBox grounding lifts Qwen3-VL-8B to 48.46% on OmniSpatial, top among open models, and leads zero-shot on SpatiaLab.

What problem this solves

LVLMs recognize objects reliably but fall apart on spatial relations: which side of the girl the lamp sits on, which object a driver meets first. The newest fix, following OpenAI's “Thinking with Images”, has the model emit bounding boxes for task-relevant objects inside its reasoning trace. The training signal is the weak point. Rewards check only the final answer, so a correct answer reinforces the entire trajectory, including localization the model itself is unsure about. Figure 1 makes this concrete: both models answer “back-right” correctly, but the baseline's candidate boxes scatter across the image while SpatialCORE's cluster tightly around its picks. A right answer does not imply confident grounding, and that gap is usable training signal.

Method

SpatialCORE is a post-training framework built on GRPO, applied to Qwen3-VL-8B-Thinking with LoRA. Rewards have three parts (answer, format, spatial); the spatial term is the contribution:

• Reference boxes. Before training, GPT-4o-mini extracts entity phrases from the question and options, filtering non-boxable relational terms like “left” or “distance”; Grounding DINO detects them zero-shot. Detected boxes serve as pseudo-GT, with the detection score stored as a validity weight. A manual audit of 262 boxes found 95.8% correct.
• Scoring. Each predicted box scores against pseudo-GT boxes with 0.8 × clipped IoU plus 0.2 × bag-of-words label cosine similarity, scaled by validity; Hungarian matching then picks the best one-to-one assignment. The label term handles wording gaps like “black sedan ahead” versus “black sedan”.
• Confidence. Box coordinates are generated digit token by digit token. Per-digit full-vocabulary entropy, normalized by log|D| and averaged over the four coordinates, gives a box's uncertainty H. The weight ω = 0.1 + 0.9 × (1 − H) rewards boxes that are both accurate and emitted with certainty; the 0.1 floor keeps high-uncertainty boxes slightly positive.
• Coverage. Match quality and soft recall over pseudo-GT boxes combine in an α=2 harmonic mean that slightly favors coverage, so trajectories localizing only one object cannot win.

An answer gate couples the three: full spatial reward when the answer is correct, scaled by 0.3 when wrong, so useful grounding on wrong-answer trajectories keeps partial credit and localization can improve before answers do.

One practical detail: GRPO requires a 1,000-sample SFT cold start to establish the grounded format. Skip it and the grounding reward decays to zero while the model stops emitting boxes entirely; with it, 95–100% of responses keep the format through training. Total cost: 2 H100s for about 31 hours.

Results

Trained on OmniSpatial (8.4K questions, 50 subcategories) and evaluated on its held-out split; zero-shot on SpatiaLab (1.4K questions, naturalistic scenes):

Among proprietary references, the weakest on OmniSpatial is GPT-4.1-mini at 48.87 with Gemini-2.5-flash at 53.16; on SpatiaLab, GPT-4o-mini scores 46.50 and Claude 3.5 Haiku 42.93.

The same-backbone comparison is the cleanest: +2.54 points in-distribution and +1.70 zero-shot over standard GRPO, the net contribution of confidence weighting. Margins over specialized models are larger (+7.37 over VST-RL-7B, +8.04 over SpaceThinker), though those opponents are mostly 3B-scale.

Ablations on the spatial interaction subset: full model 56.33; removing confidence weighting drops it to 52.33, the largest single drop; removing vision LoRA 53.00, pseudo-GT validity 53.33, answer gate 54.67.

Confidence itself becomes calibrated: after training, samples in the lowest coordinate-entropy quartile reach 77% matched IoU versus 36% in the highest, and the entropy-IoU correlation flips from +0.38 to −0.42. Retraining with 40% of pseudo-GT boxes deleted, relabeled, or displaced still yields 46.70, above the 43.90 base.

Why it matters

Prior RL methods in this line reward whether grounding is correct; SpatialCORE adds a third dimension, how certain the model is. The signal is free: coordinate-token entropy is already there, with no human labels or extra models. For anyone post-training LVLMs this is a plug-in reward-shaping idea, at 62 H100 GPU-hours with LoRA, code released. The honest read: a 2.54-point net gain over same-backbone GRPO is incremental, and part of the SOTA headline comes from a strong base model and a size advantage over 3B specialized opponents.

Limitations

The paper draws its own boundary: only the BBox post-training objective changes, so depth, multi-view geometry, and non-boxable spatial concepts stay out of scope. Results confirm it. Allocentric and hypothetical subcategories gain least, and Size & Scale on SpatiaLab barely moves, since single-view 2D boxes carry no evidence for those tasks.

The supervision source deserves scrutiny. The localization “ground truth” comes from Grounding DINO; 71% of boxable samples get a valid reference, and the rest train on answer and format rewards alone. The 95.8% audit precision and the 40%-corruption retraining experiment show the pipeline is reliable, but the reward ceiling is capped by the detector.

Against closed models, 48.46 on OmniSpatial still sits below the weakest proprietary reference, GPT-4.1-mini at 48.87, so the “reaches proprietary range” framing is generous. On SpatiaLab it genuinely beats GPT-4o-mini and Claude 3.5 Haiku.

Terms

• Generated grounding: the model outputs bounding boxes or other localization as part of its own reasoning trace.
• GRPO: Group Relative Policy Optimization, an RL algorithm that samples a group of trajectories per prompt and normalizes rewards within the group to estimate advantages.
• Pseudo-GT BBox: reference box produced offline by an external detector, used as the localization target in reward computation.
• REC (Referring Expression Comprehension): localizing an object in an image from a natural-language phrase.
• IoU: intersection over union, the standard overlap measure between two boxes.
• Coordinate-token entropy: full-vocabulary Shannon entropy of the digit tokens emitting a box's coordinates, normalized; measures how certain the model is about that box.

Source

• arXiv 2609.38716v1
• PDF
• Hugging Face
• GitHub · https://github.com/rafiibnsultan/SpatialCORE

Related papers

• RealMe adapts 14B FLUX so eight 720p views reconstruct 2K volumetric video
• Tencent 4B VLM rejects absent objects at 67.4% while unifying boxes and mattes
• LayerEditor loses just 1% SSIM when the edit is rebuilt from its RGBA layers
• GAFU compresses generation features into 256 tokens and beats same-data SFT in 15 of 15 tests
• LaLa-World plans at 59.4 vs RAEv2's 48.0 and runs 13.9× faster
• FAPL lifts motion order from 48% to 61% and factor-edit accuracy to 58%

All paper explainers