Detect LLM hallucinations in 1.3µs on CPU — but 120B models hallucinate with unanimous false certainty
More_Slide5739 · reddit · 2026-10-02
Local-model users often detect hallucinations with Oxford's Semantic Entropy (Nature paper), which needs an NLI cross-encoder doing pairwise comparisons — costly on consumer GPUs. The author built Spanda (Rsc), a zero-dependency Python metric: normalize strings from 5 samples at T=0.7 and compute exact-match Shannon entropy on pure CPU in 1.3 microseconds, zero GPU cost. Benchmarked on GSM8K/TriviaQA across 1.5B–120B models:
- 1.5B (Qwen), AUROC≈0.58: small models format too erratically for exact-match clustering;
- 7B (Mistral), AUROC≈0.71: matches the heavy DeBERTa NLI judge (0.706 vs 0.705);
- 27B (Qwen), AUROC≈0.89: when the model knows an answer it emits identical tokens across stochastic paths; otherwise it branches into diverse wrong answers;
- 120B collapses to AUROC 0.09: "Confident Mode Collapse" — it hallucinates the exact same wrong answer across all 5 runs with zero entropy, defeating even NLI-based detection.
Practical takeaway: for 7B–27B models on structured tasks (math, code, JSON, SQL, discrete QA), 5-path exact-match entropy gives 0.89 AUROC with no neural guardrails. Code, logs and full writeup are open source on GitHub (Spnda).
More from Infra
- Samsung reportedly quoting mid-to-high $4/Gb for HBM4, over 3x the $1.50/Gb price of HBM3E — zephyr_z9 · 2026-10-02
- Dev open-sources Bobcat, a local inference engine claiming fastest LLM runs on Apple Silicon Macs — Available_Pressure47 · 2026-10-02
- Toshiba to invest ¥60B to double AI data center HDD capacity by fiscal 2027 — zephyr_z9 · 2026-10-02
- HeteroFold Enables Prefill-Free Cross-Family KV Cache Transfer, 10.7x Faster at 32K Context — UniversityofSouthernCalifornia · 2026-10-02
- Zuckerberg: Multi-Gigawatt Training Clusters Can Brute-Force Their Way to AGI — rohanpaul_ai · 2026-10-02
- CodexBar: open-source menu bar app showing AI coding quotas (22k stars) — lxfater · 2026-10-02