40B MoE with 2.5B active params: 4-bit lm_head trick costs only 0.7 pts
AIFlow_ML · x · 2026-10-22
- AIFlowML details a searched architecture: 49 layers, 256 experts (6 per token plus 1 shared), 35 sliding-window layers (513 tokens) and 14 full-attention layers with no positional encoding; the search moved from 35B/3B active to 40B/2.5B active.
- Quantization trick: with only 2.5B active params, the lmhead dominates bytes read per token. Eureka put it in 4-bit "against common practice" and recovered accuracy via self-distillation from the 16-bit model, with NVFP4 landing within 0.7 points.
Related event: Eureka Machine: 40B MoE activates just 2.5B, with 4-bit lm_head(3 posts)→
More from Infra
- Deutsche Bank Deploys Farsight On-Prem as Financial AI Question Shifts to Where Models Run — huggingface · 2026-10-22
- Gradio distills FLUX.3 Klein's 9B prompt upsampler into 0.8B and 2B models that run on CPU — Gradio · 2026-10-22
- Securing coding agents with Docker Sandboxes and edge LLM inference at ATO — HowDevelop · 2026-10-22
- Revolut's massive outage blamed on GCP, but ex-sales contact says it rejected multi-zone redundancy — thedealdirector · 2026-10-22
- 0.9B Model Transcribes a 4-Person Meeting on a MacBook With 0.08% Speaker Confusion — Binary_orchid · 2026-10-22
- Qdrant 1.20 ships BM25 hybrid search without sparse vectors — qdrant_engine · 2026-10-22