Axolotl v0.21.0 ships 501B MoE QLoRA fine-tuning on a single 8xH200 node
ethayarajh · x · 2026-10-17
Open-source fine-tuning framework Axolotl released v0.21.0 (59 commits since v0.20.0), headlined by:
- Reflection Beam fine-tuning: the 501B-parameter MoE with 23B active params can now be QLoRA-tuned on a single 8xH200 node, with FSDP2 and devicemap configs plus expert-parallel LoRA recipes for B200 in bf16, NVFP4, and NF4.
- Native diffusion LM training: a top-level diffusionlm: block fine-tunes Nemotron-Labs-Diffusion 3B/8B, Dream v0, and DiffusionGemma without plugins, plus Typed-Decision Training on diffusion LMs.
- Multimodal preference optimization: DPO, KTO, and GRPO now accept image datasets.
- Speedups: eager LoRA is 8% faster; CoreWeave cloud training supported.
The previous v0.20.0 added GGUF exports, NVFP4 LoRA, and long-context training.
More from Research
- Perplexity decider v1.2 lands #3 on Decision Index as longer training erodes backbone skills — antoine_chaffin · 2026-10-17
- Perplexity traces decider v1.2 regressions to longer training, mitigates via model souping — antoine_chaffin · 2026-10-17
- Packing multiple questions into one forward pass saves compute but adds instability — antoine_chaffin · 2026-10-17
- pplx-decider packs multiple questions per state into one bidirectional forward pass — antoine_chaffin · 2026-10-17
- Dual-mask fine-tuning recovers cacheability with only -0.4 DI vs full bidirectional — antoine_chaffin · 2026-10-17
- Perplexity ships pplx-decider v1.1: bidirectional attention adds +2.8 DI, new sizes 27B/4B — antoine_chaffin · 2026-10-17