Spanda: Open-source Rust engine does sub-microsecond LLM hallucination gating at 760ns
liquidngas · hn · 2026-09-12
Developers open-sourced Spanda, a Rust-based sub-microsecond engine and gateway for LLM epistemic uncertainty quantification and hallucination gating, with a Python SDK (pip install spnda).
Key idea
- Semantic Entropy (Nature 2024) detects hallucinations well, but its quadratic NLI cross-encoder bottleneck (90ms GPU overhead with DeBERTa) makes it unusable for real-time production serving.
- The authors found that exact-match normalized entropy (Rsc) achieves the same discriminative AUROC on structured reasoning (GSM8K) while running in under 1 microsecond on CPU (760 nanoseconds), with no GPU needed.
Benchmarks vs Python proxies
- Kernel latency: 760 ns (Rust) vs 15 ms (Python)
- Memory RSS: 3 MB vs 230 MB
- Cold start: 3.6 ms vs 1.17 s
The authors are answering questions on HN about the Rust implementation, the zero-GPU math, and benchmark methodology.
More from Infra
- SemiAnalysis: OpenAI reportedly negotiating ~1GW TPU commitment while seeking sites — dylan522p · 2026-10-28
- I tested 11 video upscaling methods for AI video: local, Topaz and cloud compared — lopas99 · 2026-10-28
- Thermo-backed CTO says AI is in its dial-up phase, touts disaggregated compute for an ultrafast 'broadband moment' — beffjezos · 2026-10-28
- DeepSeek-V4.1-Flash Q2 runs at 41 tok/s on one RTX 3090, 90% token match vs API — musicurgy · 2026-10-28
- USearch v3 RC ships after 1,000 days, its libraries now embedded in 550K-star projects — viglovikov · 2026-10-28
- SK Hynix says memory demand will outpace supply through 2030 — Chemical_Hawk_6307 · 2026-10-28