Spanda: Open-source Rust engine does sub-microsecond LLM hallucination gating at 760ns
liquidngas · hn · 2026-09-12
Developers open-sourced Spanda, a Rust-based sub-microsecond engine and gateway for LLM epistemic uncertainty quantification and hallucination gating, with a Python SDK (pip install spnda).
Key idea
- Semantic Entropy (Nature 2024) detects hallucinations well, but its quadratic NLI cross-encoder bottleneck (90ms GPU overhead with DeBERTa) makes it unusable for real-time production serving.
- The authors found that exact-match normalized entropy (Rsc) achieves the same discriminative AUROC on structured reasoning (GSM8K) while running in under 1 microsecond on CPU (760 nanoseconds), with no GPU needed.
Benchmarks vs Python proxies
- Kernel latency: 760 ns (Rust) vs 15 ms (Python)
- Memory RSS: 3 MB vs 230 MB
- Cold start: 3.6 ms vs 1.17 s
The authors are answering questions on HN about the Rust implementation, the zero-GPU math, and benchmark methodology.
More from Infra
- DeepInfra launches DeepCluster dedicated B300 clusters from $2.99/GPU-hour — niloofar_mire · 2026-09-12
- OpenAI's storage platform Habitat grew 10x YoY; Python service peaked at 20M requests/sec — xeophon · 2026-09-12
- Tencent's Open-Source CubeSandbox v0.7 Ships 60ms-Cold-Start MicroVMs for Agents — dr_cintas · 2026-09-12
- Dynamic llama.cpp Config Manager Pushes 27B Model From 167k to 262k Context on One 32GB GPU — wadeAlexC · 2026-09-12
- DigitalOcean Launches M.A.R.S. Managed Agent Runtime With First-Party OpenAI Agents API Support — OpenAIDevs · 2026-09-12
- Instinct may burn $100M+ a year in tokens, and open-weight models aren't actually cheaper — ivan_bezdomny · 2026-09-12