SpanUQ probe flags uncertain spans in LLM answers with 0.910 F1
jiqizhixin · x · 2026-11-01
Researchers from Amazon, Northeastern University and collaborators present SpanUQ at NeurIPS 2026: a 25M-parameter probe that localizes which parts of a generated answer deserve scrutiny.
- Reads hidden states from a frozen LLM; a DETR-style decoder uses learned queries to locate meaningful text spans
- A mixture-of-Beta head predicts per-span uncertainty, achieving 0.908–0.944 AUROC across five backbones
- Training signals come from multi-sample verification; the deployed probe skips that expensive sampling per request
- More localized than a single answer-level confidence score; low predicted uncertainty doesn't guarantee truth, and each backbone needs its own probe
More from Research
- HuggingFace releases FineDNA dataset, turning Qwen3-14B into a DNA model that beats GPT-6.1 — loubnabnl · 2026-11-01
- If your AI scores 100%, your eval is wrong — an AI safety researcher's eval playbook — AI Engineer · 2026-11-01
- Control Vectors experiment: letting an agent manage its own activations — sloppenheimer · 2026-11-01
- FlashRSI open-sources RSI research: 12 algorithms, 30+ benchmarks, OOD gains still elusive — HuaxiuYaoML · 2026-11-01
- Claude designed scorpion toxin binders: 17 of 211 lab-tested designs bound — shoumikgoswami · 2026-11-01
- Carbon-X: midtrained Qwen3-14B beats every open LLM on DNA benchmarks — LoubnaBenAllal1 · 2026-11-01