ExLlamaV3 Official Release Brings Major Speedups
Unstable_Llama · reddit · 2026-07-15
ExLlamaV3 has officially released version 1.0.0. The author summarized a massive batch of performance and engineering upgrades over the past year.
Key updates include:
- Removing flash-attention-2 and xformers dependencies
- Expanding tensor parallel support to more models, including Gemma4
- A new attention kernel: supporting online cache quantization, SWA layers, and attention sinks
- A new graph path, conv1d kernel, INT8 GEMV kernel, and MoE kernel scheduler
- Massively improved GEMM/GEMV performance on the Ampere architecture
- Added support for multiple new models, along with tons of bugfixes / QoL improvements
- Optimized extension compilation speed
The author's positioning is clear: this is a major release focused on raw inference performance, especially suited for users interested in local deployment and high-throughput inference.
More from Infra
- DeepSpeed shows AutoEP/AutoTP/AutoSP parallelism gains, hosts office hours — PyTorch · 2026-10-27
- Cloudflare Durable Objects validated as top architecture for AI agent WebSocket servers — tristanbob · 2026-10-27
- Gary Marcus: With Q3 data added, Anthropic's compute-efficiency growth shows deceleration — GaryMarcus · 2026-10-27
- SpaceX locks in $40B GPU financing deal with Apollo, the largest ever committed — TechCrunch AI · 2026-10-27
- Inference is just Kubernetes with a multi-trillion-dollar market, and the communities overlap — matt_slotnick · 2026-10-27
- vLLM eyes Rust migration as inference engines race to rewrite the stack — AccBalanced · 2026-10-27