OneTrans: Unified Feature Interaction and Sequence Modeling with One Transformer in Industrial Recommender
Zhaoqi Zhang, Haolei Pei, Jun Guo, Tianyu Wang, Yufei Feng, Hui Sun, Shaowei Liu, Aixin Sun
WWW 2026)
cs.IR
2025-10-30
OneTrans jointly models user sequences and feature crosses in one Transformer; versus RankMixer+Transformer, Feeds GMV/u +5.68% at −3.91% p99 latency.
Industrial ranking has scaled on two separate tracks. Feature-interaction modules (Wukong, RankMixer) grow crosses among static profile, context, and candidate fields. Sequence modules (DIN, Transformers, LONGER) compress clicks, carts, and purchases into a user vector. The default graph is encode-then-interaction: fold the history into one vector, concatenate static features, then cross.
Each track scales. Information does not flow both ways. Static features cannot reshape sequence states during encoding, and the two modules cannot share KV cache or FlashAttention. ByteDance's e-commerce ranker had already moved from DCNv2+DIN to RankMixer+Transformer. The remaining bottleneck was the two-stage split itself.
OneTrans puts sequence modeling and feature interaction in one causal Transformer.
A unified tokenizer turns multi-behavior histories into S-tokens and user/item/context fields into NS-tokens, with learned [SEP] markers between sequences. Time-ordered interleaving wins when timestamps exist; otherwise sequences are concatenated by intent strength. On the non-sequential side, Auto-Split (one projection, then slice) beats hand-grouped tokens.
Each OneTrans block is a pre-norm causal Transformer with mixed parameters: all S-tokens share one QKV and FFN, each NS-token has its own. Heterogeneous static features collapse attention if they share a projection, so RMSNorm is applied as pre-norm and NS-tokens get private weights. The causal mask lets each S-token see earlier events only; each NS-token sees the full history and preceding NS-tokens.
A pyramid schedule shrinks the S-token query set by layer while keys and values stay full-length, so information drains into the tail and the NS-tokens. Candidates in one request share the user sequence: S-side runs once, NS-side runs per candidate, and sequence KV can be appended across requests. FlashAttention-2, mixed precision, and activation checkpointing sit on top.
OneTransS is 100M: 6 layers, width 256, 4 heads, sequential queries 1190→12. OneTransL: 8 layers, width 384, queries 1500→16.
Offline logs: about 29.1 billion impressions, 27.9 million users, 10.2 million items. Versus DCNv2+DIN:
| Model | CTR AUC / UAUC | CVR AUC / UAUC | Params | TFLOPs |
| RankMixer+DIN | +0.27% / +0.36% | +0.43% / +0.19% | 107M | 1.31 |
| RankMixer+Transformer | +0.57% / +0.90% | +0.52% / +0.75% | 109M | 2.51 |
| OneTransS | +1.13% / +1.77% | +0.90% / +1.66% | 91M | 2.64 |
| OneTransL | +1.53% / +2.79% | +1.14% / +3.23% | 330M | 8.62 |
On an unoptimized OneTransS (407 ms train step, 54 ms p99), stacking pyramid, cross-request KV cache, FlashAttention, and mixed precision cuts p99 by up to 69.1%. OneTransL versus DCNv2+DIN: 8.62 vs 0.06 TFLOPs, 330M vs 10M params, 13.2 vs 13.6 ms p99, MFU 30.8% vs 13.4%.
Online control is RankMixer+Transformer (100M) without sequence KV cache. Feeds: click/u +7.74%, order/u +4.35%, GMV/u +5.68%, p99 −3.91%. Mall: GMV/u +3.67%. Active days +0.75%. Cold-start product order/u +13.59%.
Recommender scaling does not have to stay "a crossing module times a sequence module." One causal Transformer is what lets KV cache and FlashAttention apply to the whole ranker, and what lets static features rewrite behavior states from layer one. For teams already on RankMixer, this is the next hop, not another widening of the crossing stack. In their calibration, +0.3% offline CTR AUC usually moves online metrics; OneTransL's +1.53% is a large step on that scale.
The authors say scaling past OneTransL is still bound by online efficiency. Candidate-specific sequences (SIM-style) are pooled into NS-tokens and cannot use the shared S-side cache. Full attention matches causal attention offline; causal wins because of KV cache, not accuracy. The data are ByteDance e-commerce logs, with no public table. Pyramid lengths are a heuristic rounded to multiples of 32; that schedule is not searched. Both arms trained on 1.5 years of production data, so numbers will shrink on products with shorter windows.