Open-Source Rust/CUDA Engine reflex Adds DeepSeek MLA, Cold-Start 1.3x Faster Than llama.cpp
SaveAmerica2024 · reddit · 2026-09-24
A developer open-sourced reflex (MIT), a GGUF-native Rust/CUDA inference engine optimized for cold-start latency (process launch to first token) rather than steady-state throughput.
- Model support: full Multi-head Latent Attention for DeepSeek-V2/V3 (verified against the real DeepSeek-V2-Lite checkpoint), plus dense Qwen3, Qwen3-MoE, and Qwen3.5 hybrid Gated DeltaNet.
- Performance: all kernels AOT-compiled by nvcc at build time (no NVRTC JIT); 1.3-1.4x faster cold-start than llama.cpp on an A6000; vLLM was 24-52x slower cold (noted as an unfair comparison).
- System1 mode: single-pass candidate scoring with probabilities instead of autoregressive decoding — useful for agent decision loops picking among known options.
- Positioned for embedding/one-shot exec (C-FFI + Python bindings), batchsize always 1, GPU-only.
More from coding & agent
- Setting Astra's Thinking to 'Extra High' Makes It Over-Engineer Unit Tests — astralmatrix · 2026-09-24
- Free O'Reilly Book Offers a Pragmatic Framework for Scaling AI in Engineering Teams — blaizedsouza · 2026-09-24
- Zilliz CTO: agents make the enterprise data layer impossible to ignore — No_Engineer_1224 · 2026-09-24
- Running Android emulator + Chrome with 60fps streaming in a $0.072/hr cloud VM for always-on agents — cem2ran · 2026-09-24
- 10 agent reruns reached the right neighborhood, none reproduced the key observation — rohanpaul_ai · 2026-09-24
- Using the Jev model for offensive security: OPSC ranking and sensitive file detection in Mythic — dyn___ · 2026-09-24