Measuring factuality in AI search: sentence-level auto-raters and the FACTS benchmark
lilyraynyc · x · 2026-11-06
At TechSEO Connect, Anfal Saddiqui explained that factuality is getting harder to measure in AI search: auto-raters now evaluate sentence by sentence, checking each claim and whether the sentence actually appeared in the grounding context.
The slides outline the FACTS Benchmark Suite scoring: an Eligibility check (off-topic answers count as inaccurate), a Grounding check (every unsupported claim counts as inaccurate), then averaged judge scores from Gemini 2.5 Flash and GPT-5, with support levels from fully supported to no answer. Highly relevant to SEO/marketing teams navigating AI search.
More from Research
- ICLR reviewer flags novelty overlap with arXiv paper that drew from the authors' own workshop work — BlancheMinerva · 2026-11-07
- Optimistic Hedge regret is O(n log n) for 2 players but exponential for 3, new paper shows — roydanroy · 2026-11-07
- Exo: Berkeley's agent runtime that edits its own harness, cutting model cost 35-61% — dair_ai · 2026-11-06
- Symbolic belief revision plus geometric projections lifts RAG reranking to 0.481 nDCG@10 — letandrewcook · 2026-11-06
- Charter: A Lean 4 DSL for Formally Verified Rules Governing AI Agents — QuixiAI · 2026-11-06
- ICLR 2027 sees 42,790 submissions; 57% of reviews are a 2, 2-2-3 most common — CSProfKGD · 2026-11-06