Blind Men and the Elephant: Probing the Epistemic Myopia of LLMs under Long-Tail Divergent Knowledge
Zhuoshi Pan, Junru Lu, Yan Qian, H. Vicky Zhao, Di Yin, Xing Sun
cs.CL
2026-08-29
ElephantBench turns low-exposure web disagreements into 1,094 closed-book multi-account questions. Kimi-K3 fully recalls both accounts on 52.38% of items; the rest are almost all one-sided. Minority-side exposure tracks completeness better than majority-side exposure.
Standard factual QA assumes one canonical answer. A model can emit the popular date and score as correct while the other sourced date is missing from parametric memory. Long-tail QA still grades a single gold. Knowledge-conflict suites are mostly open-book: they hand the model the disagreeing passages and test reading, not memory.
ElephantBench asks two closed-book questions: can the model recall a low-exposure fact at all, and can it recover every verified account of the same subject-attribute pair. The running example is Mother Teresa's birth date: 26 August 1910 in one source, 27 August in another, both publicly attested.
A web corpus is split with a DCLM fastText quality classifier. The probe mines the discarded low-score slice, where rare facts are more likely to live. Documents become a graph: support edges link agreeing accounts of one fact, conflict edges link incompatible values for the same pair. To avoid all-pairs LLM calls, candidates are retrieved by knowledge-point clusters plus an entity inverted index, then labeled support, conflict, or none.
Each conflict edge expands into a local subgraph and yields a named-entity question and a clue-based twin that share the same answer set. Validation has three gates: an LLM checks source support, a web agent seeks independent authoritative evidence, and human reviewers drop entity mismatches, answer leakage, and question-fact errors. From 4,127 subgraphs come 8,254 candidates; 1,094 items remain across 22 fields.
Evaluation is closed-book. GPT-5.6-Sol grades complete, partial, or failed recall and reports conditional completeness K. Exact agreement with human raters is 90.13%-93.36%, Cohen's κ 0.815-0.877.
Among 32 configurations, open-weight Kimi-K3 leads with 52.38% complete recall, 45.25% partial, and 2.38% failed. Gemini-3.1-Pro sits at 50.37%, GPT-5.5 at 50.18%. The top three almost always recover at least one verified value (failed recall 2.19%-2.65%). The main failure is incompleteness, not blanks.
Open-weight models under 10B average 5.48% complete recall; the largest open system reaches 52.38%. Qwen3.5 from 2B to 397B lifts complete recall from 1.65% to 32.27% and cuts failed recall from 81.35% to 8.50%, while partial recall rises to 59.23%. Reasoning adds 13.99 points for GPT-5.6-Sol and 12.89 for GPT-OSS-120B, and slightly hurts Qwen3.5 2B/4B.
A one-standard-deviation rise in majority-side documents tracks +14.18 points of partial recall and -10.17 failed. The same rise on the minority side tracks +15.13 complete and -15.41 partial. People, organizations, and events reach 38.7% complete recall; consumer products and services sit at 6.2%. An oracle over all 32 configurations climbs from 52.4% to 81.2% complete, yet 206 items (18.8%) have no complete recall from any model.
Single-answer leaderboards score one-sided memory as success. If a product must handle disputed dates, prices, or rulings, complete recall is the closer risk metric. On the data side, extra minority-source exposure is more associated with completeness than more majority exposure. The graph pipeline can be reused on other low-exposure corpora; code and data are Apache 2.0.
This is a diagnostic probe, not a training method. It locates a memory hole. It does not fill it.
Pretraining mixes are private. Exposure counts come from a public web corpus and are associations, not measurements of what any given model saw. 1,094 items across 22 fields do not exhaust the tail, and disagreements are mostly two values in one slot. The 32-model pool is a snapshot. Each retained account has independent public support and can still inherit source error. The release is for factual-recall research only.