SonicBoom: A Python-Free C++ Runtime Beats llama.cpp Decode by 14% on a Laptop 3050
nalaginrut · reddit · 2026-10-13
Developer open-sourced SonicBoom (v0.0.1, GPL-3.0), a pure C++/CUDA tensor runtime with a C ABI and zero Python, targeting local LLM inference on consumer GPUs.
Numbers (Gemma 4 E4B-it Q3KM, laptop RTX 3050 8GB, same GGUF as llama.cpp):
- Decode: 51.65 tok/s vs 45.20 tok/s (+14%)
- Decode @ depth 2048: 47.13 vs 45.21 tok/s
- Prefill: 1051 tok/s, roughly half of llama.cpp (1944)
- TTFT with 4000-token prefix reuse: 503 ms
Why it matters: the quantized matmul kernels are ported from ggml—identical to llama.cpp's. The 14% decode lead comes entirely from the graph compiler layer above them: activations quantized once and reused, norm/activation fused with quantization, dead computation removed, and each decode step captured and replayed as a single CUDA Graph. ggml-derived code is only 8% of the core.
Limitations: prefill is a real batched path (int8 tensor cores + cuBLAS) but still slow; only Gemma 4 E4B verified; only Q3K/Q4K/Q5K formats (most HF Q4KM files won't load); tested on sm86 only; greedy sampling only. Roadmap: MLIR GPU codegen, speculative decoding, Qwen support.
More from Infra
- Real-time voice agent model survey: teams cut LLM latency from 1s to 0.3s by switching off GPT-4.1 — Natural-Chemical1513 · 2026-10-13
- Prediction: Opus 5.5 will run locally on a single 24GB GPU within 8 months — TheZachMueller · 2026-10-13
- Colibri: pure-C inference engine runs 2.8T-parameter MoE models by streaming experts from disk — tom_doerr · 2026-10-13
- Cloud vs On-Prem Is the Wrong Question for AI Agents: Split Workloads by Component — DavidLinthicum · 2026-10-13
- With RTX 5090 Discontinued, ComfyUI Users Hunt for Alternative GPUs — Impressive-Scene-562 · 2026-10-13
- Dev builds own peering network hitting 350Mbps in the wild, agent makes the landing page — vincent_koc · 2026-10-13