EXL3-QLoRA lets you fine-tune 2-8bpw quantized models, beating BNB NF4 on loss
Unstable_Llama · reddit · 2026-10-12
Reddit user UnstableLlama released exl3-qlora, a native ExLlamaV3 training implementation (plain PyTorch, no transformers) that enables QLoRA fine-tuning on 2-8bpw EXL3 quantized models, breaking free of the old BNB 4-bit lock-in. EXL3 offers the highest precision at a given size, and adapters save in standard PEFT format for TabbyAPI/PEFT or merging back to base.
Key features
- Model support: Llama3, Mistral Small/Medium, Qwen3 dense/MoE, Gemma3/4 (incl. MoE), Arcee Trinity — each verified against inference forward
- SFT/DPO/KTO, rsLoRA, PiSSA, optional LoRA on MoE experts/embeddings/LM head
- Single GPU, multi-GPU layer split, DDP; sample packing, jinja chat rendering with tool calls; gradient checkpointing, 8-bit/paged optimizers, Liger kernels; browser-based live run monitor and real-time training while serving chat
Benchmarks (Qwen3.5-4B, same data/hardware): EXL3 6.0bpw eval loss 2.128 (12.7 GiB peak), 4.0bpw 2.152, 3.0bpw 2.165 vs bnb NF4 (axolotl) 2.183. bnb is 2x faster, but exl3 is more flexible, VRAM-efficient, and yields lower loss. Several models trained with this pipeline are already released on Hugging Face.
More from coding & agent
- Hu Yanbin built a stage light-beam instrument using ByteDance's TRAE coding tool — vista8 · 2026-10-12
- JetBrains paper: coding-agent benchmarks should match real users' task flows — rohanpaul_ai · 2026-10-12
- Dev is building an AGPL-licensed self-hostable Overleaf alternative — AlpinDale · 2026-10-12
- mitsuhiko's weekend 'software factory' with his kids shows how good AI coding models have gotten — mitsuhiko · 2026-10-12
- agent-seat: an MCP server that gives your computer-use AI its own Windows desktop session — Fun-Instruction-3430 · 2026-10-12
- yes-if: PayPal hackathon project turns payment risk 'no' into the cheapest conditional 'yes, if' — damnGruz · 2026-10-12