AnswerMap: Faithful Spatial Interpretability of VLMs from Answer Posteriors
Mohamed Eltahir, Fardows Adam, Duaa M. Tahir, Lama Alamoudi, Sana Ammar, Atheer A. Alboloshi, Jory Albluey, Tanveer Hussain, Naeemullah Khan
cs.CV, cs.LG
2026-09-28
AnswerMap builds a map from yes/no band posteriors. It hits the model's point at AUC 0.85 vs 0.38 for attention; deleting its region flips 53% of correct answers vs 19%.
When a VLM answers a visual query, the natural check is spatial: which part of the image the answer came from. A radiologist reading a generated report needs that check. So does an auditor. Current tools fail in different ways.
Text rationales share the output head but the wrong modality. A sentence names no region, cannot be tested against the image, and often fails to track what drove the prediction. Attention and gradient relevancy sit in the right modality, yet they originate earlier than the answer. On VLMs the distortion is worse: deep-layer attention piles onto sink tokens that the text does not need, and image tokens get a vanishing share. Occlusion is visual, black-box, and reads the final output, but it scores one cell at a time at O(K^2) cost, and each score is only the drop from an almost-certain full-image answer. If the rest of the image covers for the cell, the cell scores nothing.
What is missing is a cell score that is itself an answer about the cell, read from the output head, at linear cost.
AnswerMap cuts the image into K row bands and K column bands. Each band is shown alone to the frozen model with one yes/no question: is the query in this band. Generated text is ignored. The first-token logits for Yes and No become a yes posterior. The outer product of the row and column vectors is a KxK query-conditioned map. A cell lights up only if both the row and the column through it say yes with confidence.
Default K=8 costs 16 forward passes; occlusion at the same grid costs 65. Bands are cut in pixel space, so dynamic resolution, tiling, and token merging stay invisible.
A single grid locks one band width. Coarse bands carry context; fine bands are local. Coprime grids such as {3,5} are probed separately, upsampled to a shared 64x64, min-max normalized, and multiplied. A location survives only if every bandwidth endorses it. Coprime boundaries do not coincide, so K1+K2 queries yield K1+K2-1 resolution pieces.
The map is the rationale. The answer is a fixed read-out R computed from the map. Expectation is a weighted mean of cell centres after subtracting the map floor, a continuous sub-cell coordinate. The maximum is grounding strength. The top-mass region becomes a crop fed back to the model.
Every strip lacks an index and the surrounding image, and the question is the same. A language prior can raise or lower the whole map but cannot move mass across it. Any location a read-out computes comes from the strips.
Primary model: Qwen3-VL-4B-Instruct, replicated on 8B, 30B-A3B, and InternVL3.5-8B. All frozen, greedy, no training. Baselines include best-layer attention (layer swept on 200 RefCOCOg examples per model, then frozen; those 200 sit inside the official 800, which can only help attention), T-MM relevancy, TAM, and same-grid occlusion.
Test 1: does the map land where the model points.
| Method | RefCOCOg NSS | RefCOCOg AUC |
| AnswerMap {3,5} | 1.56 | 0.853 |
| AnswerMap K=8 | 1.36 | 0.824 |
| Occlusion (65 queries) | 1.04 | 0.649 |
| TAM | 0.77 | 0.693 |
| T-MM | 0.33 | 0.656 |
| Best-layer attention | -0.22 | 0.380 |
| Random map | -0.05 | 0.486 |
A blank image lands at chance. A swapped image follows the swapped image and has no correlation with this image's point. Attention stays at or below chance across all 36 decoder layers; its expectation barely moves with the query.
Test 2: delete the map's 8/64 top-mass cells (about 12.5% of the image, filled grey) and re-ask questions the model got right.
| Method | TextVQA flip | GQA flip |
| AnswerMap | 53.4% | 22.4% |
| Occlusion | 48.8% | 18.2% |
| T-MM | 36.8% | 17.1% |
| Best-layer attention | 18.8% | 3.0% |
| Uniform random region | 8.2% | 4.5% |
After labeling items by whole-image deletion, the probe region flips 57.4% of image-used questions and 22.0% of unused ones. Across 4B to 30B and InternVL, Test 1 AUC sits between 0.768 and 0.824, Test 2 flip between 45.8% and 53.4%. The same operator runs on GPT-6-sol through its API from first-token logprobs only.
Three read-outs scored against ground truth. On Qwen3-VL-4B RefCOCOg, the generated point hits 94.5% point-in-region and the map expectation 67.5%; generation is the richer channel. On medical Lingshu-7B, pointing fails to parse on 30% of queries and the order flips: expectation 57.7%, generated point 46.7%. The maximum separates grounded from hallucinated yes claims among 4,006 POPE yeses at ROC-AUC 0.833, with 16 queries and no labels. Deleting the map region flips 69.5% of hallucinated yeses and 41.0% of grounded ones; random deletion is 11.2% vs 1.4%. Cropping the top-mass region and appending it on TextVQA moves accuracy from 91.6% to 92.4%; a random crop drops it to 89.0%. The map crop corrects 50% of 42 wrong answers, the random crop 38%, while disturbing 4% vs 6% of answers that were already right.
Ablations: single-grid NSS peaks at K=6 (1.46), falls to 1.14 at K=8, and collapses to 0.22 at K=40. Coprime {3,5} reaches 1.50 at the same 16 queries and stays flat across a five-fold budget. On one A100, 0.41s per map at 4B and 1.25s at 30B, against 1.80s and 5.90s for occlusion.
If the model exposes first-token logprobs, a spatial map that can be checked against the image is available with no weight access and no training. Closed APIs are in scope. Under this protocol, attention as a spatial explanation falls below chance. Uncalibrated per-example attention is noise on both tests.
The read-out turns the explainer into an output interface. Coordinates need not be stuffed into discrete tokens. Hallucination screening need not wait for a generated paragraph. A crop repairs half of the wrong answers. The cost is 16 forwards per map: slower than internals, four to five times cheaper than occlusion. The useful settings are audits, medical image checks, and models whose native pointing fails to parse. This is a better interface, not a new training recipe.
Two limits are stated in the paper. The operator needs non-square inputs, because each band is a long thin strip. The map knows which rows and columns contain the query, not which pairs do: two separate matching objects also light the empty cells where their rows and columns cross. That is the geometry of a rank-1 outer product.
Test 1 scores against the model's own point, so faithfulness is relative to the model's belief, not to annotated boxes. When Qwen3-VL-4B already points well, the expectation trails the generated point by about 27 points on RefCOCOg. The attention layer is chosen on 200 examples that also sit in the evaluation set; the paper says this can only favor attention, and attention still loses. The TextVQA crop net gain is 0.8 points; repairing half the errors and lifting net accuracy are different claims. The POPE AUC 0.833 is computed on claims the model already answered yes, not on a full detection task. K=8 is not the best single grid; K=6 has higher NSS.