Shallow LLM guidance beats deep injection in VQA: 39.4% vs 31.8% on OK-VQA
Aneerban Saha · hf · 2026-10-15
Independent researchers challenge the assumption that deeper LLM knowledge integration helps VQA. Using Llama 3.3 70B to generate structured reasoning guidance, visual focus concepts, distractors and external knowledge, they compare two strategies in a frozen ViT + FLAN-T5 setup on OK-VQA:
- Shallow late fusion: 39.4% exact match, +10.0 over the no-guidance baseline
- Deep attention-matrix injection: only 31.8%
Integration depth must match the data regime — shallow fusion wins under data-scarce fine-tuning. They also release an LLM-based augmentation pipeline that expands training data over tenfold via question paraphrasing.
More from Research
- AAAI/ACM AIES paper interviews 15 data scientists on why responsible AI practice is hard to sustain — ChinasaTOkolo · 2026-10-15
- Schmidt Sciences awards $10M to 19 AI humanities research teams — OdedRechavi · 2026-10-15
- Reddit debate: is overfitting killing LLM creativity? — Expensive-Paint-9490 · 2026-10-15
- Mistral and CERN partner to build a Large Physics Model trained on billions of simulated collisions — MistralAI · 2026-10-15
- DataFlex turns LLM training data strategy into four composable, mid-run-adjustable ops — Puzzleheaded_Box2842 · 2026-10-15
- Global Transport coupling lifts guided flow generation: ImageNet FID 2.06 to 1.91 — mmbronstein · 2026-10-15