POPE benchmark audit: yes/no readout choice alone shifts VLM F1 by 6.13 points
Kesav Kumar Jayakumar · hf · 2026-10-20
A NeurIPS 2026 workshop paper systematically audits yes/no extraction in POPE hallucination evaluations:
- In 9,000 greedy answers, LLaVA-1.5-7B never generated the tokens compared by the common two-token readout (yes 3582 / no 1217) — it always emits Yes(3869)/No(1939). Reading through the emitted tokens, the same answers score 6.13 F1 points higher on POPE adversarial (0.7608→0.8221) with no change in model behavior.
- Five of six surveyed papers report adversarial gains smaller than this readout shift, and none states its readout protocol.
- Porting LLaVA token IDs to Mistral/Qwen vocabularies yields 0.00 F1 — the confound is protocol-wide, not model-specific.
- Proposed fix: derive yes/no token sets from each model's tokenizer, log generated token IDs, verify sets under the exact prompt template, and release prediction-level records.
More from Research
- Conditional Program Shrinks LLM Lookup Tables 75% While Improving Fidelity — dnotitia · 2026-10-20
- Ablation study: domain-scoped editing worth up to 3.7 points in HARNESS² agent framework — morgymcg · 2026-10-20
- Indie researcher stores documents as SSD "memory cartridges" for a frozen Qwen2.5-72B, claims byte-identical replay — Nearby_Indication474 · 2026-10-20
- Survey compares Vision Encoders across 9 frontier models, from SigLIP 2 to encoder-free — liuziwei7 · 2026-10-20
- Critic-only RL gains momentum in continuous control, new follow-up work — m_wulfmeier · 2026-10-20
- Open-Science: open-source workbench turns coding agents into auditable research assistants — rohanpaul_ai · 2026-10-20