Maxwell-V1-Flash: 35B MoE with 3B active params, 256K context, 6.7x faster attention
pmttyji · reddit · 2026-11-17
Maxwell Team released Maxwell-V1.0-Flash on Hugging Face: a MoE LLM with 35B total parameters (3B active per token, excluding embeddings) and 256K context.
Key techniques
- Maxwell Block Sparse Attention (MBSA): each KV group keeps 8 prefix and 8 local 128-token blocks plus up to 16 remote blocks chosen by a learned indexer, bounding attention at 4,096 tokens per group while keeping the full-history KV cache; the kernel is 6.7x faster than dense GQA at 256K.
- Manifold-Constrained Hyper-Connections: multiple residual streams with learned, input-dependent aggregation, widening information flow without extra attention/FFN compute.
- Fine-grained MoE with always-active shared experts, dynamic routing correction bias, and sequence-wise load-balancing loss.
Training: 26T multilingual tokens, Muon optimizer on most hidden weights; dense start, block-sparse attention introduced at 128K mid-training, staged extension to 256K. Script-aware tokenizer stays compact for low-resource Southeast Asian languages (Thai, Lao, Myanmar, Khmer, Shan).
Post-training: two-stage all-domain SFT, then separate RL branches for general, verifiable reasoning (math/science/logic/code), and agentic tasks (search, terminal, function calling, SWE), merged via multi-teacher on-policy distillation.
More from Models
- miHoYo unveils Gamer, a generalist AI agent that plays games by watching the screen — minchoi · 2026-11-17
- Codex Pro's $200 plan includes $200/month in API credits, saving devs real money — burkov · 2026-11-17
- Claim: Claude covertly halted a game dev's GPU deployments after refusing a clone job — RexDouglass · 2026-11-17
- NVIDIA open-sources Nemotron-Labs-Sigma: continuous diffusion LLMs in 14 checkpoints at 3B/8B — ArashVahdat · 2026-11-17
- Fully AI-Written Text Is Instantly Recognizable to Model Users, Argues Analyst — Afinetheorem · 2026-11-17
- Claude designed three working devices; the 25-degree board tilt detail stands out — nukebuddy · 2026-11-17