ExLlamaV3 Official Release Brings Major Speedups
Unstable_Llama · reddit · 2026-07-15
ExLlamaV3 has officially released version 1.0.0. The author summarized a massive batch of performance and engineering upgrades over the past year.
Key updates include:
- Removing flash-attention-2 and xformers dependencies
- Expanding tensor parallel support to more models, including Gemma4
- A new attention kernel: supporting online cache quantization, SWA layers, and attention sinks
- A new graph path, conv1d kernel, INT8 GEMV kernel, and MoE kernel scheduler
- Massively improved GEMM/GEMV performance on the Ampere architecture
- Added support for multiple new models, along with tons of bugfixes / QoL improvements
- Optimized extension compilation speed
The author's positioning is clear: this is a major release focused on raw inference performance, especially suited for users interested in local deployment and high-throughput inference.
More from Infra
- Anthropic S-1 chart fuels positive-ROI narrative on AI infrastructure buildout — BenBajarin · 2026-10-26
- Nearly half of Claude revenue now flows through cloud marketplace commitments, up from 11% — BenBajarin · 2026-10-26
- Anthropic ARR goes vertical after price cuts: tokens up 4x, $100B+ ARR back in play — GavinSBaker · 2026-10-26
- Alex Banks argues power caps GPU supply, real risk is demand cooling first — thealexbanks · 2026-10-26
- Analyst: Ant has the most fungible AI workload of any model provider today — BenBajarin · 2026-10-26
- Anthropic S-1: training infra edge cut per-task cost 40% and doubled revenue growth — BenBajarin · 2026-10-26