Trained 35 domain LoRAs on a MoE with MLX on one Mac Studio — the base model beat most of them
Holiday_Poetry_5133 · reddit · 2026-11-08
A Reddit user trained 35 domain-expert LoRA adapters on Qwen3.6-35B-A3B (256-expert MoE, 3B active) using MLX on a single Mac Studio M3 Ultra 512GB in BF16, releasing all adapters, datasets, configs, router and training scripts under Apache-2.0.
Key lessons:
- Fusing MoE LoRAs into 8-bit produced garbage; only runtime BF16 adapter hot-swapping worked (one base loaded + per-request swaps, 2s).
- A MiniLM router over 34 domains tops out at 46% top-1 / 70% top-3 — routing is the weak link, not the adapters.
- Negative result: on an independent audit bench the base 35B model beat most specialist adapters; only HTML/CSS clearly improved. The author now routes almost everything to the base.
- mx.random.normal isn't bit-exact across Apple GPU generations (M1 vs M3+); reported upstream and a docs PR was merged into mlx.
They're asking whether anyone has made domain LoRAs beat a strong MoE base at 30B scale without full-parameter training.
More from Infra
- antirez: most local inference is single-user, just attach pi or opencode to local server — antirez · 2026-11-08
- antirez: large batches switch to layer-major inference, nearly as fast as full-resident — antirez · 2026-11-08
- Vitalik backs anti-data-center protests: algorithmic progress makes bigger compute unnecessary — paulnovosad · 2026-11-08
- AI costs falling faster than any past tech, says analyst chart — oran_ge · 2026-11-08
- Forget Lambos, This AI Practitioner Just Wants an NVIDIA Vera Rubin — AIFlow_ML · 2026-11-08
- Samsung, SK Hynix and Micron in talks to standardize fab robots for fully automated fabs by 2030 — zephyr_z9 · 2026-11-08