When audio, video, text disagree: controlled study of cost-aware acquisition vs abstention
NRC-CNRC · hf · 2026-10-14
National Research Council Canada studies when to acquire a costly third score or abstain when audio, video, and text disagree, via a controlled three-score benchmark where abstention is rewarded only under a designated ambiguity mechanism and penalized under mixed corruption.
Findings:
- A threshold policy matches always-request decisions with fewer requests; its edge over always-answer fusion depends on the ambiguity reward
- On a partially held-out synthetic split, the threshold policy reaches 0.789±0.006 targeted decision accuracy and 0.481±0.014 utility across 83 seeds, vs 0.626/0.252 for a three-score majority reference that uses more information
- In matched-budget tests, a train-only value selector beats no-query and matched-random policies at 10% and 25% budgets but lowers utility at 50% and 63.7%
- If all abstentions count as incorrect, majority outranks the threshold policy
Takeaway: full-modality accuracy and pre-request selection value are distinct, and selection value depends on budget and the observed-pair ranking.
More from Research
- Chollet: Ideal AI-for-Science Outcome Is Millions of AI-Aided Scientists, Not a Few Rich Labs — fchollet · 2026-10-14
- Feeling a dipole's electric field along an ellipse: 2.08kq at the end, 0.13kq on top — TinfoilTricorn · 2026-10-14
- Alibaba's Representation Guidance: plain pixel-diffusion Transformers beat bigger models — bdsqlsz · 2026-10-14
- Visualizing the Fourier series of a sawtooth wave, with Gibbs overshoot hitting ~9% — TinfoilTricorn · 2026-10-14
- Gemma Embedding 2 hits SOTA on Vietnamese speech after 3,000 hours of audio, 2 hours on 8xH100 — stablequan · 2026-10-14
- KILVO: robust multimodal odometry for humanoid robots fusing encoders, IMU, LiDAR and vision — rsasaki0109 · 2026-10-14