Inherit4Rec: Parameter Inheritance for Efficient Scaling of Recommendation Models
Ruihao Zhang, Bo Chen, Xiao Wang, Jinlong Jiao, Tijian Hu, Qinglin Jia, Xiuqiang He, Xiangyu Zhao, Chaoyi Ma, Ruiming Tang, Wenwu Ou
cs.IR
2026-09-20
Kuaishou's Inherit4Rec widens a trained recommender with exact forward preservation or converts it to sparse MoE, within 0.12% GAUC of retraining at 72% lower cost.
The standard move for industrial recommenders is to train a bigger dense model. At Kuaishou the accounting is stark: a 0.3B UniFormer trained from scratch to convergence costs more than 250,000 GPU hours, and capacity upgrades happen repeatedly, so every rebuild burns money while production sits on a suboptimal model. Sparse MoE cuts serving compute to a fraction, but converting a dense checkpoint to MoE normally means retraining from zero and writing off the sunk cost. Parameter inheritance (net2net, bert2BERT, LLaMA-MoE) reuses old weights in new structures, but those methods assume static corpora, and recommendation distributions drift daily, so transplanting them causes visible drops.
Inherit4Rec defines two transformations, both on SwiGLU FFNs.
D2D, dense to denser. The gate and up projections are replicated c times, widening d to (c+1)d, while a newly sampled set of down projections is centered so the replicas sum to zero. At the moment of expansion the new network's forward output exactly equals the old one's, so nothing is lost; the zero-sum down rows also break gradient symmetry, letting replicas differentiate during training. Asymmetric training keeps inherited parameters on their original Adam moments and LR schedule while new parameters start from zero moments with their own warmup-and-decay schedule.
D2S, dense to sparse. A forward-only calibration pass over current traffic scores every channel by mean squared contribution: the top-m channels form an always-on shared expert, and the rest are grouped into R routed experts by co-activation similarity (capacity-constrained greedy seeding plus swap refinement). The router initializes fresh, a Switch-style load-balancing loss keeps experts evenly used, and each token computes only the shared expert plus one routed expert, about a quarter of the original width.
KuaiRand-1K plus an industrial Kuaishou short-video dataset, four prediction objectives each, scored by GAUC:
Industrially, D2D grows the model from 28M to 74M parameters (FFN width 256 to 1024) and D2S activates a 1:4 ratio. Ablations: removing the zero-sum constraint costs 0.066 to 0.169 percentage points, dropping optimizer-state transfer costs 0.093 to 0.254, and removing co-activation partitioning costs 0.132 to 0.240, the largest single factor.
These margins look small, but a thousandth of GAUC is a perceptible business difference in industrial ranking, and every comparison continues training on the same data stream, so the denominator is real GPU hours.
This is a cheaper path for teams that upgrade capacity repeatedly: the expanded model holds its old performance on day one and improves from there, skipping the window where a fresh model underperforms the one it replaces. D2S moves an already-paid dense checkpoint into a serving structure at a fraction of the compute. Nothing in the mechanism is recommendation-specific beyond the non-stationary data; SwiGLU FFNs are standard LLM parts, and the same recipe applies to continued pretraining on other drifting distributions.
Only per-token FFNs are covered; attention and other modules are untouched. The paper validates a single transformation, with repeated chained D2D and D2S evolution left as future work. Exact forward preservation is mathematically impossible under sparse routing, since only a subset of channels is evaluated. The industrial conclusions ride on Kuaishou's traffic shape, and the public-dataset scale tops out at 302M parameters, still far from very-large-model territory.