Meta’s GAMUT benchmark scores long answers on missing facts, and the best model gets 58.7%
rohanpaul_ai · x · 2026-07-24
Meta proposes GAMUT to measure what factual answers leave out
Meta’s new paper argues that factuality is not just about avoiding wrong claims — it also means including the important facts an answer should contain.
- GAMUT introduces a two-level rubric for evaluating open-ended generation.
- The framework first builds a structured meta-rubric covering required facts, acceptable choices, ordered steps, relationships, and importance.
- That structure is then compiled into machine-checkable pass/fail rubrics that an LLM judge can score more consistently.
- The benchmark includes 1,813 questions grounded in wearable imagery across 10 domains.
- Across 14 strong models, the best score was only 58.7%; missing information caused more failures than false statements.
- Meta says the setup is modality-agnostic and also released a text-only variant.
More from Research
- Diamandis: AI Cancer Navigator and $100 Genome Sequencing Close the Care Gap — PeterDiamandis · 2026-10-27
- EMNLP Oral: training-free Nexus Sampling cuts KV-cache memory 10x; OpenResearcher tops BrowseComp-Plus at 54.8% — yuz9yuz · 2026-10-27
- ICML 2026 position paper: LLM 'thinking traces' are a harmful anthropomorphism — rao2z · 2026-10-27
- Forking gap follows a power law in context frequency; masking rare n-grams shrinks it — ZimingLiu11 · 2026-10-27
- How forking happens: n-gram tables split logits into non-interacting subspaces — ZimingLiu11 · 2026-10-27
- The best LoRA explainer video yet: rank constraints meet intrinsic dimension — sophiamyang · 2026-10-27