Successfully Fine-Tuning 35B MoE: Insights into Model Routing
VectorInst · x · 2026-07-30
Vector's team shared their successful setup for fine-tuning a 35B multimodal MoE on 4 A100s: keeping MoE layers intact while stacking 3 optimizations—4-bit quantization, Flash Attention 2, and a forward hook reducing logit memory from 23.3GB to 0.3GB.
With the model running, they audited the learned routing: layers 1-28 route by modality (audio, video, text), while layers 29-48 shift to semantic routing. Text tokens were highly dynamic, showing a clear shift from perceptual encoding to higher-level reasoning in deeper layers.
Related event: Vector Institute Shares 35B MoE Fine-Tuning Practices(3 posts)→
More from Research
- Relay-OPD: A Novel On-Policy Distillation Framework That Fixes Prefix Failure by Teacher Takeover — _akhaliq · 2026-07-30
- Compute Surge: 10 Major Scientific Breakthroughs AI Could Unlock by 2028 — Annual_Judge_7272 · 2026-07-30
- Inside SOTA Deep Research: Native Model Training and 150 Sub-Agents — SimonShaoleiDu · 2026-07-30
- Princeton Prof Reviews MIT Nonconvex Optimization Paper: Prize Remains Open — HazanPrinceton · 2026-07-30
- OpenDerm: An Open-Source 4-DOF Home Robot for Early Skin Cancer Detection — plopesresearch · 2026-07-30
- Deep Learning Research Course Updates: Deep Dive into Multi-class Softmax Loss — alfcnz · 2026-07-30