Towards a Densing Law for User Representation Learning at Billion-Scale Capacity
Bin Dou, Junru Zhang, Zhaoyi Yuan, Wuliang Huang, Letian Gong, Baokun Wang, Huan Li, Yu Cheng, Weiqiang Wang
cs.IR, cs.AI
2026-08-24
On Alipay PayBill logs, raw user embeddings saturate past 30M users and 60-day windows. ALGN uses 63% of RQ-VAE capacity and lifts classification AUC to 76.43.
Industrial user encoders are usually scaled the obvious way: more users, longer histories, bigger Transformers. Payment logs fight that recipe. Daily spend is repetitive, so volume grows much faster than new task-relevant signal.
Ant Group's DeepFind team and Zhejiang University measured the stall on Alipay production data sized for billion-user capacity. Downstream probes flatten once the user pool passes about 0.03B, the history window passes about 60 days, or the encoder passes about 0.2B parameters. A 0.4B encoder can still drive the contrastive pretraining loss lower. Linear-probe accuracy does not follow. The bottleneck is information density in the raw event stream, not parameter count.
Pretraining splits each user's timeline into past and future. A Transformer encodes the past. A LoRA-tuned Qwen3-Embedding model encodes a templated text of the future. InfoNCE aligns the two. Evaluation freezes the encoder and runs three probes: classification on 50 scenario sets, plus text retrieval and user-to-user retrieval on 22 sets each, about 0.5M users per set.
To raise density they first compress long PayBill histories with RQ-VAE. Early residual codebooks capture coarse consumption categories and routines; later ones keep merchant-level leftovers. The same user encoder is trained on the discrete codes, so gains can be blamed on denser inputs rather than a new architecture.
They then fit a configuration rule they call the Behavioral Densing Law: the log of the smallest tokenizer capacity that still hits near-peak utility grows roughly linearly with the log of data scale (users and days). The slope depends on the tokenizer and the source. VQ-VAE is steepest because similar semantics occupy separate full IDs. RQ-VAE is milder through compositional reuse. SARQ is flattest. Within one tokenizer, the slope also tracks the square of intra-source uniqueness, measured as mean k-NN cosine distance of LLM text embeddings.
ALGN applies the same idea per snippet. At each residual level a gate looks at residual norm (how much is still unexplained) and path uncertainty (how ambiguous the current code path is). Continuation rises with those two signals and falls with marginal code cost. A geometric length prior with γ=0.3 stops the model from always using the maximum depth.
Raw scaling hits a wall. Adding users from 0.01B to 0.03B still helps; 0.03B to 0.1B barely moves the needle. Extending history from 30 to 60 days helps; 60 to 120 days does not. Growing the encoder from 0.05B to 0.2B improves probes; 0.2B to 0.4B keeps lowering training loss while test accuracy stalls. Contrastive loss only falls from about 8.28 to 8.15 as the window goes from 30 to 180 days, and curves for D≥90 days nearly overlap.
Tokenized inputs pull ahead exactly where raw modeling saturates, around 64-day windows and 1.2×10^7 users. A PayBill compression recipe transferred to SPM and MiniProgram keeps about 95.0% and 97.5% of source-matched AUC.
ALGN is compared at 180 days and 0.1B users on PayBill classification:
| Method | Capacity | AUC | KS | Acc |
| RQ-VAE | 100% | 74.56 | 39.02 | 82.44 |
| SARQ | 76.24% | 75.36 | 41.28 | 83.16 |
| ALGN | 63.47% | 76.43 | 43.31 | 83.52 |
Against SARQ that is +1.07/+2.03/+0.36 AUC/KS/Acc at 13.23 points less capacity. The fitted densing slope drops to about 0.59.
If a ranking or risk stack is still buying "another six months of logs" or "another 2× parameters," this paper's operational claim is narrower: past those thresholds, tokenizer capacity is the better spend. After a one-time calibration, a cheap uniqueness statistic can size the next codebook. ALGN then spends residual depth on ambiguous periods and stops early on routines.
Treat it as an industrial recipe, not a universal scaling law. Public-benchmark confirmation is still pending.
The authors flag a single platform and a single modality; video-style behavior is untested, and public-dataset checks are unfinished. The raw scaling grid only reaches 0.1B users, short of the 2B figure in the notation table. Tokenized-vs-raw plots report crossover points without a table of absolute AUCs at each scale. ALGN's headline table is PayBill classification only; retrieval is waved through as "similar." Calling a log-linear fit a law oversells a platform-specific calibration.