IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts
Ran Cheng, Longfei Xu, Zheng Liu, Kaikui Liu, Xiangxiang Chu
cs.LG
2026-09-18
IntBMoE composes the full expert pool into a bounded codebook of blocks and routes each token to a few of them, reaching 73.76% ImageNet Top-1 and +2.4% UVCTR in AMap serving.
Mixture-of-Experts wants more capacity per token without paying for every expert. Existing designs couple three quantities that should be independent: participation (how many experts contribute knowledge to a token), execution (how many expert networks actually run), and materialization (how many expert-sized parameter sets must be built and stored).
Sparse routing keeps execution cheap and also shrinks participation: unselected experts neither affect the output nor receive a gradient from that token. Dense output mixing restores full participation and makes compute grow with the expert count. Parameter merging runs a single composed expert, but every distinct routing unit needs its own composed copy, so memory tracks the number of routing decisions.
The design question is whether every token can use the full pool while execution stays sparse and the number of materialized experts stays bounded.
IntBMoE splits construction from execution and replaces the Transformer FFN. Each module holds a learned codebook of K input-independent embeddings. A shared hypernetwork maps each embedding to unconstrained value and gate coefficients. At every internal layer those coefficients mix a shared expert-basis pool, with a 1/sqrt(E) scale, into K reusable L-layer blocks. Negative coefficients are allowed, so composition spans the linear hull of the bases rather than their convex hull.
A two-layer ReLU MLP then picks Top-k blocks per token. Before a block runs, the token is concatenated with that block's codebook vector and passed through a sigmoid feature filter, so different blocks see different slices of the same token. Inside the block, Dual-Path Residual Gating multiplies a value path by a residual SiLU-gated path after RMSNorm. Routed outputs are mixed by softmax weights and added to an always-on shared SwiGLU.
Because blocks depend only on the codebook, they can be composed once and cached. After caching, request-time compute no longer grows with pool size E.
The main study trains an 8-layer DeiT-Tiny-style backbone from scratch on ImageNet-1K at a 24M parameter budget, averaged over three seeds. Default hyperparameters are (K, E, k, L) = (8, 16, 2, 2).
| Method | Top-1 | Top-5 | Activated (M) | FLOPs (G) |
| Dense | 66.40% | 87.69% | 3.94 | 1.45 |
| SMEAR | 71.78% | 90.33% | 22.89 | 1.49 |
| Expert Choice | 71.70% | 90.30% | 5.14 | 1.96 |
| IntBMoE | 73.76% | 91.48% | 23.11 | 4.06 |
| IntBMoE (cached) | - | - | - | 3.46 |
That is 1.98 points above the strongest baseline, SMEAR. Dropping the gate path falls to 68.04% Top-1; collapsing each two-layer block into a parameter-matched single layer falls to 68.78%. Removing any examined expert basis hurts accuracy; deleting E15 in layer 0 costs 9.15 points.
On MiniPile, test loss is 2.6802 and PPL is 14.59, 2.9% below μMoE (CP) at 15.03 and 12.4% below the dense backbone at 16.66. On IntTravel, HR@1 is 0.6852 versus 0.6837 for μMoE (TR).
In AMap's opening-screen POI recommender, cached IntBMoE ran a one-week A/B at about 5,000 QPS: 19 ms mean, 38 ms P99, inside a 60 ms budget, with a 2.4% relative UVCTR gain. The control was the production model without an MoE module.
This is a rare MoE paper that reports both academic numbers and a production A/B under a hard latency cap. Caching takes composition off the request path, which is the part recommendation systems actually pay for.
The ImageNet study is still a 24M from-scratch vision model, not an LLM-scale test. The online control is "no MoE" rather than "swap in a strong sparse MoE."
The 1.98-point gain over SMEAR comes with FLOPs rising from 1.49G to 4.06G (3.46G cached). Activated parameters are near the full budget because the whole pool is used to build blocks. The IntTravel HR@1 edge is 0.0015, small enough to sit in noise. Growing K from 8 to 32 adds 0.07 points; growing E from 16 to 64 adds 0.15. The public code points at DreamX-Rec, without a standalone training recipe detached from the recommender.