A Glance Is All You Need: Single-Pass Fine-Grained Image Captioning with SimLoss
Suryaansh Jain, Rahasya Barkur, Vishal G, Ryan Rossi, Franck Dernoncourt, Jack Wang, Koustava Goswami, Nedim Lipka, Puneet Mathur, Samyadeep Basu, Seunghyun Yoon
cs.CV
2026-09-01
SimLoss FFT, trained with InfoNCE on frozen image embeddings and no captions, hits 0.8485 precision and 0.7023 F1 on IIW-400, matching CapMAS at ~20× lower A100 latency.
Today's vision-language models write fluent captions that still miss the attributes that make one image unlike another: material, texture, count, spatial layout. "A lamp on a table" can be true and still drop the ceramic urn base and the spiral-bound notebook beside it.
The usual fix is more inference. CapMAS samples several captions, splits them into atomic claims, checks each claim against the image, then rewrites from the survivors. Quality goes up. Cost goes up with it. On an A100 the pipeline takes 115.31 seconds per image; a single-pass Qwen2.5-VL-7B caption takes 8.66 seconds.
Supervision is also the wrong grain. Sampled COCO captions average 10.0 words. Human ImageInWords descriptions on IIW-400 average 171.2 words, about 17 times longer. Distilling CapMAS outputs is not a clean substitute: after processing, 7.0 of 29.3 atomic propositions are judged false, a factuality ratio of 0.766. Imitating that teacher copies roughly a quarter of its unsupported claims.
The question is whether a captioner can learn to keep those details at training time, then run in one forward pass, without fine-grained human captions and without pipeline pseudo-labels.
SimLoss treats a detailed caption as a lossy channel that should still identify its source image. A generic sentence is compatible with many similar photos. A caption that keeps materials, counts, and spatial relations should retrieve the source more reliably. Mutual information I(image, caption) is the ideal; it is hard to maximize over discrete tokens, so the paper moves the same idea before decoding.
A frozen Qwen3-VL-Embed encoder writes an image embedding. A trainable Qwen2.5-VL-7B, given the same image and the prompt "Describe this image in detail.", mean-pools its hidden states and maps them through a two-layer MLP into that embedding space. InfoNCE then treats the matched pair as the positive and every other image in the batch as a negative. Gradients update only LoRA adapters and the projector. Base weights and the embedding model stay frozen.
No caption text is required. COCO images are used; COCO captions are thrown away. Because the LoRA weights are shared with the generation path, the contrastive loss reshapes the representation the decoder will read, without dictating wording. At inference the encoder and projector come off, leaving the adapted VLM in a single pass.
Two instantiations cover different access to the teacher:
Baselines cover four families: plain Qwen2.5-VL-7B, five-stage CapMAS, FeedQuill PPO with a composite of unit-level F1 plus CLIP plus CIDEr, and PAPO with random or YOLO-guided masking.
Evaluation follows CapMAS on IIW-400. Precision is the fraction of atomic propositions a GPT-4o judge marks as supported by the image and the IIW reference. Recall is accuracy on human-verified multiple-choice questions answered from the caption alone, image withheld. F1 is their harmonic mean. CLAIR (scaled to 0-1) asks an LLM whether candidate and reference captions describe the same image.
| Method | Precision | Recall | F1 | Words | A100 s/img |
| Qwen2.5-VL-7B | 0.7884 | 0.6002 | 0.6815 | 348 | 8.66 |
| CapMAS | 0.8467 | 0.6003 | 0.7025 | 190 | 115.31 |
| FeedQuill | 0.7966 | 0.5967 | 0.6823 | 164 | 8.97 |
| PAPO+YOLO | 0.7936 | 0.5969 | 0.6813 | 182 | 7.30 |
| SimLoss FFT | 0.8485 | 0.5991 | 0.7023 | 115 | 5.77 |
| SimLoss GRPO | 0.8227 | 0.6015 | 0.6949 | 133 | 6.58 |
Recall is flat: every method sits between 0.595 and 0.602. The entire F1 spread is precision. SimLoss FFT posts the highest precision at 0.8485, 0.0018 above CapMAS, with F1 0.0002 lower, while latency falls from 115.31 s to 5.77 s, a measured 20.0×. Mean length drops from 348 words to 115, and the standard deviation from 47 to 13.
SimLoss GRPO takes the best recall (0.6015) and CLAIR (0.858), with precision 0.8227 and F1 0.6949. A black-box embedding reward can nudge coverage slightly wider and also lets in plausible but less grounded detail.
Qualitatively, FFT more often writes scenes as foreground / mid-ground / background, reads "MILSPED AML" off a railcar, and names a "chainmail hauberk" and "red surcoat". In matched-coverage cases it answers the same questions in about one third as many words; extra baseline text is mostly numbered outlines, closing recaps, and symbolic readings. The same specificity backfires: both SimLoss variants call an abstract driftwood moose a deer.
For anyone who needs detailed captions in production, the recipe is concrete: train against a frozen image embedding, deploy without a verification stack. IIW-style long human captions are not required, and neither is a CapMAS teacher. On one A100 the wait goes from about two minutes to about six seconds, with slightly higher precision.
It also kills a common assumption: fine-grained does not mean long. The 348-word baseline scores 0.7884 precision; the 115-word FFT model scores 0.8485, with recall almost unchanged. The extra words were mostly idle.
This is a training-objective swap, not a new architecture. The backbone is still Qwen2.5-VL-7B; LoRA and a projector do the work. Because recall never moved, embedding alignment did not teach the model to mention more of the visual evidence humans ask about. It taught the model to write fewer unsupported claims. If the product metric is coverage rather than a short clean caption, the GRPO variant or a light post-hoc check is the better fit.
The authors say this clearly: dropping caption targets means depending on a frozen embedding teacher. That teacher's biases and visual grain decide which distinctions get rewarded. Comparing or ensembling teachers across domains, and combining FFT precision with GRPO recall, are named as future work and not run.
The flatter fact is the recall line. If fine-grained captioning means putting discriminative visual evidence into text, the human question set barely moved for any method. Near-CapMAS F1 is mostly a precision gain plus shorter, more stable prose. Calling that "recovering multi-stage quality" only holds if quality is defined as atomic-proposition factuality.
The evaluation has a circular edge. The appendix cites agreeableness bias in LLM judges (true-positive rate near 96%, true-negative rate below 25%). Precision in this paper is still a GPT-4o verdict over atomic claims. SimLoss does not train on that judge. The "highest precision" medal is still awarded by one.
All numbers are on IIW-400. Training images come from COCO. There is no reported ablation on temperature, batch size, projector depth, or a swapped embedding teacher. The moose/deer miss shows that writing more specific words is not the same as writing the right ones.