Study on Statistical Self-Consistency in LLMs
AlayaLab · hf · 2026-07-17
This work discusses the issue of **statistical self-consistency** in large language models (LLMs). If prompts are understood as conditional distribution estimates, model outputs should satisfy basic probability identities, such as the law of total probability. The authors use a **binary tree hierarchical partitioning** to divide populations into subgroups, querying the model for subgroup estimates, aggregating them back to the overall population, and comparing this with direct overall estimates. Results show that frontier models generally violate these consistency properties. The paper also identifies a phenomenon termed the **macro fallacy**: - Overall estimates reconstructed from finer-grained subgroup answers are often closer to human reference data than direct overall estimates. - This effect persists across different tree structures and estimation tasks. - It can be partially mitigated through implicit prompting. The authors conclude that models actually possess relevant subgroup knowledge but fail to stably propagate this knowledge to macro-level aggregated judgments. Therefore, **statistical self-consistency** can serve as a reference-free evaluation metric.
Related event: Study Reveals LLMs Lack Statistical Self-Consistency(2 posts)→
More from Research
- Draft paper uses Markov-chain eigenfunctions to build partitions and speed up sampling — michaelchchoi · 2026-07-21
- Autoresearch proposes packaging ML runs as studies with questions, analysis, and code diffs — morgymcg · 2026-07-21
- GitHub repo adds lightweight ternary QAT for Prism-ML Bonsai models — terminoid_ · 2026-07-21
- Qdrant co-hosts a Munich meetup on search, retrieval, and agentic RAG on July 23 — qdrant_engine · 2026-07-21
- GigaChat Audio targets long-form audio grounding with timestamps across 120-minute inputs — ai-sage · 2026-07-21
- Paper models Transformer components as stochastic geometry and tests five architectures — Zhihua Liang · 2026-07-21