RankMixer: Scaling Up Ranking Models in Industrial Recommenders
Jie Zhu, Zhifang Fan, Xiaoxie Zhu, Yuchen Jiang, Hangyu Wang, Xintian Han, Haoran Ding, Xinmin Wang, Wenlin Zhao, Zhen Gong, Huizhi Yang, Zheng Chai, Zhe Chen, Yuchao Zheng, Qiwei Chen, Feng Zhang, Xun Zhou, Peng Xu, Xiao Yang, Di Wu, Zuotao Liu
cs.IR
2025-07-21
RankMixer replaces handcrafted feature crosses with token mixing and per-token FFNs, scaling Douyin ranking from 16M to 1.1B params at unchanged latency; duration +1.08%.
Ranking models in production recommenders want to scale like LLMs, under a very different budget. Latency sits in the low tens of milliseconds, QPS is extreme, and cost tracks machine count. Most feature-crossing modules still look like the CPU era: DCN, factorization machines, hand-built crosses glued together. On GPUs they are memory-bound. Model FLOPs Utilization (MFU, realized floating-point work as a fraction of hardware peak) often sits in single digits. Douyin's live ranking baseline had 15.8 million dense parameters, 4.47% MFU, and 14.5 ms latency. Widening or stacking that design yields small, sometimes negative, gains.
RankMixer asks a concrete question: can dense parameters grow by two orders of magnitude without raising serving latency.
Features are grouped by business semantics, concatenated, then sliced into T fixed-width tokens. Too many tokens starve each one of capacity; too few collapse the model into a large MLP. The 100M setup uses T=16, width 768, 2 layers; the 1B setup uses T=32, width 1536, still 2 layers. Offline quality tracked total parameter count across width, depth, and token count, similar to LLM scaling. Wider GEMMs raise MFU, so the chosen configs grow width rather than depth.
Each RankMixer block has two stages.
For higher ROI they swap each per-token FFN for a Sparse MoE. Uniform top-k routing wastes experts on low-information tokens. RankMixer uses a ReLU gate plus an L1 penalty so high-information tokens activate more experts, and trains dense while serving sparse (DTSI) so experts do not starve. At 1/8 activation this stack nearly matches 1B dense AUC and about doubles throughput.
Offline training uses two weeks of Douyin logs: hundreds of features, trillions of daily records. A 0.0001 AUC lift is treated as significant.
| Model | Finish AUC relative | Dense params | FLOPs/batch |
| DLRM-MLP | 0 (base 0.8554) | 8.7M | 52G |
| Wukong | +0.29% | 122M | 442G |
| RankMixer-100M | +0.64% | 107M | 233G |
| RankMixer-1B | +0.95% | 1.1B | 2.1T |
On Skip, the 100M model is +0.86% AUC / +1.33% UAUC; the 1B model is +1.25% / +1.82%. Dropping token mixing costs 0.50% AUC; sharing the FFN costs 0.31%.
Online, RankMixer-1B replaced a 16M DLRM+DCN stack: about 70× parameters, 20.7× FLOPs, 3.6× lower FLOPs-per-parameter, MFU 4.47% to 44.57%, plus fp16. Latency moved from 14.5 ms to 14.3 ms. Douyin main app: active days +0.29%, duration +1.08%, finish +1.99%, like +2.39%; low-activity users +1.74% active days. Ads: AUC +0.73%, advertiser value ADVV +3.90%. A reverse A/B over eight months had not yet saturated.
The paper turns "ranking models do not scale" into an arithmetic identity: 3.6× fewer FLOPs per parameter, 10× MFU, 2× peak from fp16, and a 70× model fits the old latency envelope. For production ranking, the reusable pieces are token mixing instead of attention, per-subspace FFNs, and raising utilization before chasing a scaling law. The quality lift is incremental. The deployment is full-traffic Douyin Feed.
Sequence modeling is still a sidecar (the paper cites LONGER). RankMixer only replaces the dense crossing stack. Tokenization depends on hand-built semantic groups; a bad grouping hits quality directly. The 10B MoE path is an offline sparsity curve; what shipped is 1B dense. All numbers are Douyin logs, with no public benchmark, so transfer evidence is the ads test alone. The larger lift on low-activity users may also mean the old baseline was weaker on cold start, not that the new model "understands" the tail.