ExLlamaV3 3bpw local Flash benchmarks: 1500 tps prefill on a single RTX 5090
youcloudsofdoom · reddit · 2026-09-21
A Reddit user shares impressive results running Flash locally with ExLlamaV3/exl3 at 3bpw, saying it's replacing vllm/llama.cpp for them:
- 3x RTX 3090 + 128GB DDR4: 1500 tps prefill, 80 tps decode
- Single RTX 5090 + 128GB DDR4: 1500 tps prefill, 29 tps decode
- Both tested at 262k context with vision and speculative decoding enabled
- Quant quality at 3bpw looks good so far; a 4bpw comparison is planned
Recommended for anyone who was sleeping on exl3.
Related event: ExLlamaV3 3bpw Quantization Delivers Impressive Local Flash Performance(2 posts)→
More from Infra
- Agent Substrate Calculator Estimates Session Resume Speed Under Load — pjausovec · 2026-11-12
- Dell expands AI Data Platform: agents have a data-context problem, not a model problem — DavidLinthicum · 2026-11-12
- SGLang Megakernel Pushes Muse Spark to 1,109 tok/s, 2.1x Over Baseline — hsu_byron · 2026-11-12
- Potion UI's local login system turns your PC into a multi-user AI server — cocktailpeanut · 2026-11-12
- AWS Finally Built Its Own AWS Emulator — super_linear · 2026-11-12
- Megakernel is multiprogramming for GPUs: 'GPUs are entering the CPU age' — pranjalssh · 2026-11-12