Eureka Machine: 40B MoE activates just 2.5B, with 4-bit lm_head

Eureka Machine's discovered architecture has 49 layers, 256 experts and only 2.5B active parameters out of 40B, with an unusual 4-bit lmhead recovered by self-distillation; third-party tests show 216 tok/s on an RTX 5090, though all benchmarks are self-reported with no paper yet.

2026-10-22 ~ 2026-10-22 · 3 related posts

• Quantizing the lm_head to 4-bit: self-distillation recovers accuracy, NVFP4 within 0.7 pts — AIFlow_ML · 2026-10-22
• 40B MoE with 2.5B active params: 4-bit lm_head trick costs only 0.7 pts — AIFlow_ML · 2026-10-22
• Eureka Machine architecture: 49 layers, 256 experts, 216 tok/s on a 5090 in early runs — AIFlow_ML · 2026-10-22