llama.cpp v0.9.0 ships native FP8 quantization and NCCL multi-node tensor parallelism
victormustar · x · 2026-11-05
llama.cpp v0.9.0 is out with a batch of inference-side improvements:
- Quantization: new OCP FP8 E4M3 tensor type (saturating mode) with conversion helpers and a new ftype.
- Serving: new flashattnextrows op makes the unified KV cache efficient for multi-sequence inference, restoring non-unified KV prefill performance.
- Multi-node: ggml updated to 0.29.0, adding NCCL-based multi-node tensor parallelism plus performance work across CPU, CUDA, SYCL, Metal, Vulkan, OpenCL, and Hexagon backends.
- New architectures: Clover, Limite, Metis-1.6, and Mistral Large 4.
- Speech: Chatterbox TTS, Nemotron 3.5 Transcribe, and predefined Pocket TTS voices.
- Other: major prefill optimizations for Gemma 4 E2B/E4B, a batched tensor copy API superseding the 2d API, and server/UI fixes.
Related event: llama.cpp ships v0.9.0, v1.0.0 coming within weeks(3 posts)→
More from Infra
- Bloomberg columnist urges data center builders to read his overlooked deep-dive — asusarla · 2026-11-05
- Google launches AlloyDB Express: 2-second Postgres provisioning with scale-to-zero for coding agents — rseroter · 2026-11-05
- SGLang Summit 2026 agenda: LinkedIn serves ads model at 10K QPS, NVIDIA brings SGLang to Vera Rubin — DavidBennett__ · 2026-11-05
- A top-tier 28-minute explainer on the structural differences between GPUs, TPUs and Groq's LPU — yacinelearning · 2026-11-05
- Proving a Model's Answer Costs 70-95x the GPU Time of Running It on vLLM — petrusenko_max · 2026-11-05
- AWS shows how to plug calibrated decision models into Bedrock AgentCore Evaluations — AWS ML Blog · 2026-11-05