SonicBoom v0.0.3: compiler-first AI runtime beats llama.cpp decode by 1.18x, calls for GPU benchmarking
nalaginrut · reddit · 2026-10-19
HardenedLinux released SonicBoom v0.0.3, a GPLv3, compiler-first inference runtime with zero Python — one shared library, C ABI, Guile Scheme as first-class language. On a desktop RTX 3050 8GB vs llama.cpp -ngl 99, it hits 1.18x decode on Gemma 4 E4B Q3KM and 1.12x on Qwen2.5-0.5B Q4KM; time-to-first-token on multi-turn agent workloads dropped several-fold via prefix reuse and plan caching, with bit-identical outputs. All graph optimization runs on Boom IR, an MLIR dialect, with per-optimization ablation numbers. They admit gaps (0.98x on Qwen at depth 512, single GPU) and invite the community to benchmark via a GitHub issue, with README credit for contributors.
More from Infra
- Huawei's Atlas 960E superpod links 4,096 NPUs with near-packaged optics — bookwormengr · 2026-10-19
- OCP attendees split on CPO/NPO light sources: external CW lasers (ELSFP) vs micro-VCSEL — bookwormengr · 2026-10-19
- Robot brain inference cost fell 84% in two months, from $2.36 to $0.38 per task — victor_explore · 2026-10-19
- Cross-encoder reranking loses to a stricter cosine threshold on 2 of 3 semantic cache benchmarks — Reasonable_Royal_621 · 2026-10-19
- M5 Ultra Mac Studio rumored with 512GB RAM; 6 units chained could run 2T-parameter models locally for ~$100k — SumitGup · 2026-10-19
- Apple Silicon's memory economics: M5 Ultra offers 5.6x better $/GB/s than DGX Spark for LLM inference — SumitGup · 2026-10-19