Exploiting Feature-invariant Scaling Laws for Efficient A/B Experimentation in Recommendation Systems
Zhiliang Chen, Sammy Nammari, Jiahao Su, Xiaoqiang Luo
cs.IR, cs.LG
2026-11-10
Wukong, HSTU, and LLM recommenders keep one power law after feature changes, and five co-scaled runs (0.1-0.5, ~55% of a full train) predict a full-scale loss drop above 1%.
A new recommender feature, whether architecture, optimizer, or data, is normally judged by retraining the full model on the full dataset and comparing it with a baseline that already exists. Candidate lists in industry run into the hundreds. Giving every one of them a serving-scale retrain does not fit the compute budget or the schedule.
The cheap substitute is a small model on a data subset, dropped if the metric does not move. The sign is often wrong. In one LLM recommender, writing movie genre into the prompt lowers HR@10 on a small slice of MovieLens and raises it once the full training set is used. One small run would have killed a useful feature. Parking the probe at 0.5 scale is not something you can choose in advance: different feature families flip sign at different small scales.
The curve is still smooth. In that genre example both the baseline and the edited model fit Loss = a·N^b·D^c with R² above 0.99. The coefficients differ.
The name in the paper is feature-invariant scaling: the function family stays put when the feature changes. Helpful edits and harmful edits both sit on a power law. Only the coefficients move. Scale is s = N×D, parameter count times example count. Full scale S is the largest model actually served. When N changes, sparse embedding width, hidden width, and depth move together so the aspect ratio stays similar, which is done to keep training stable.
Baseline loss at S is treated as known. The edited model's loss at S is not. Each candidate is trained only at a few scales below S, and validation loss is modeled as the power law plus Gaussian training noise. In log space that is a linear regression, log Loss = log θ₁ + θ₂ log N + θ₃ log D. Noise variance is taken as constant across scale. Coefficients come from closed-form least squares, with n−3 in the variance denominator. Pushed to full scale, log loss has a Student-t posterior with n−3 degrees of freedom, so the interval does not need sampling. A handful of points on the left of the curve are used to place the right-hand end.
A hairline gain is the wrong shipping test. Extra compute, storage, and engineering cost eat marginal improvements. The rule: given the small-scale runs, the posterior probability that full-scale loss falls below 0.99 times the baseline must exceed 90% before a higher-fidelity run is worth paying for. The 1% bar can move with the product.
The cost math assumes co-scaling. Halving both model and data leaves about a quarter of the ND product, so one half-scale run is about a quarter of a full run. Five scales from 0.1 to 0.5 add as squares, 0.01+0.04+0.09+0.16+0.25 = 0.55, about 55% of one full training. If only data is reduced, that 55% is optimistic. When the full model does not fit on the experiment GPUs, shrinking N is what makes an A/B possible.
Power laws are an empirical choice. Other curves have no closed-form posterior and need MCMC. The reported experiments stay with the power law.
The check covers three model families and more than 40 architecture, optimizer, and data edits.
Wukong predicts clicks on Criteo. Training data is about 4.2 billion rows across 23 days; the last day, about 178 million rows, is validation. Embedding dimension grows from 8 at s=0.1 to 64 at s=1.0. HSTU predicts the next movie a user rates on MovieLens-20M, holding out each user's last interaction. At s=1.0 the model is HSTU-large, width 256, 16 blocks, 8 heads. At s=0.1 it is width 26, 2 blocks, 1 head, and the data is scaled down too. The LLM setup keeps model size fixed and scales tokens only, full-finetuning Llama-3.1-Instruct and Qwen models from about 0.5B to 8B. The prompt holds 100 candidates, so HR@10 sits higher than HSTU's all-catalog ranking. Those two HR numbers are not comparable.
Fits use validation loss. On Wukong, R² is 0.9949 for the baseline, 0.9953 with dropout, 0.9923 with SGD, and 0.9876 with an added attention layer. On Llama-3.1-8B, R² is 0.9977 for the baseline, 0.9970 with genre text, 0.9981 with ratings removed, 0.9939 with dropout, and 0.9743 with an extra layer. HSTU's baseline is 0.9989, and the other reported fits are at least 0.9972. HR@10 follows the same kind of curve.
The 1% loss cut and the 90% posterior are then applied to HSTU. For edits that do help, the probability climbs to 100% as more small scales are added. For edits that hurt, it falls toward 0. Out-of-sample error on full-scale loss shrinks as points are added. No single error percentage is stated, and coverage of the interval is not reported.
A few edits beat the published HSTU numbers on MovieLens-20M. The baseline is cited from that earlier paper. The table is HR measured at the large configuration, not the extrapolated prediction.
| Edit | HR@10 | HR@50 | HR@200 |
| HSTU-large baseline | 0.3567 | 0.6139 | 0.8073 |
| Time Gap Bias | 0.3603 (+1.01%) | 0.6186 (+0.77%) | 0.8112 (+0.48%) |
| Dropout 0.1 | 0.3590 (+0.64%) | 0.6183 (+0.72%) | 0.8107 (+0.42%) |
| Pop Debias | 0.3585 (+0.50%) | 0.6182 (+0.70%) | 0.8102 (+0.36%) |
| Wider Head 48 | 0.3584 (+0.48%) | 0.6176 (+0.60%) | 0.8097 (+0.30%) |
| RMS-Norm | 0.3584 (+0.48%) | 0.6175 (+0.59%) | 0.8094 (+0.26%) |
| Attn Residue | 0.3581 (+0.39%) | 0.6164 (+0.41%) | 0.8086 (+0.16%) |
The ranking gains are small. On HR@10 only Time Gap Bias clears a 1% relative gain. The other five land between 0.39% and 0.64%.
Crossover is common. The baseline can lead at small scale and the edit can pull ahead later, or the reverse. Genre text helps Llama and hurts Qwen. Both curves still follow a power law, and the extrapolation gets the direction right.
Full A/B retrains are expensive, and one small run often has the wrong sign. The scaling law is a filter: keep the baseline's full-scale loss, train the candidate at five scales, and read off whether full-scale loss should land under 99% of baseline, with an interval. Under co-scaling those five runs cost about 55% of one full train. If the serving model does not fit in memory, the same downscaling is what lets the experiment run.
Clearing 90% still calls for a higher-fidelity check. The MovieLens HR gains run from a fraction of a percent to about one relative percent. A 0.1% loss drop mapping to more than a 0.5% topline lift, across billions of users, is cited from the earlier HSTU paper. It is not a measurement here.
The power law breaks. On Qwen-0.5B, adding genre text lengthens context by about 30%. The curve stops looking like a power law, and the model gets clearly worse than the baseline. The stated reason is capacity: the small model cannot absorb the longer context. Once the functional form is wrong, the extrapolation and the interval fail together. The paper treats this as uncommon, but it does not say how many of the 40-plus edits broke, and it gives no residual cutoff for refusing to extrapolate.
A high R² only says the observed scales lie near the curve. Full-scale error is shown decreasing in a figure, without an MAE and without interval coverage. Probabilities that rise to 100% or fall toward 0 are plotted against edits already known to help or hurt. A hit rate for the decision rule over the 40-plus edits is not reported.
The HR table has no error bars. Most of those gains are smaller than the 1% loss gate, and there is no count of edits that would pass on loss while ranking barely moves. The public evidence is Criteo and MovieLens-20M. The LLM runs scale data only, so the 55% figure does not transfer. Latency motivates the threshold, but the decision still looks only at loss probability. Constant Gaussian noise across scale rests on training looking stable, not on a separate test.