GRACE: Generative Recommender Acceleration Engine for Real-Time Ads Retrieval
Zhou Fang, Yuhang Huang, Ang Zhang, Yihan He, Ruichao Xiao, Chao Li, Yavuz Yetim, Sibyl Yang, Xiaohan Wei, Fei Tian, Liang Wang, Liyuan Li, Nathan Yan, Gaoxiang Liu
cs.IR
2026-08-02
Generative ads retrieval emits IDs directly, bypassing pre-retrieval targeting. GRACE enforces eligible prefixes per decode step and rewrites the decoder for an 11x latency cut.
Conventional ads retrieval is a cascade: targeting rules first filter the ads eligible for a user, then retrieval and ranking take over. Generative retrieval breaks that order. A Transformer decoder autoregressively emits each ad's Semantic ID (SID, an item encoded as a short token sequence), skipping the "target first, retrieve second" step. The trouble is that advertisers set audience targeting rules (geography, age, and so on), so the eligible ad subset differs per request, while the generative model learns one fixed SID space. Existing catalog-valid constrained decoding only ensures a generated SID exists in the catalog, which is request-independent; it cannot guarantee the SID is eligible for this user. The second problem is compute: real-time ads retrieval must generate thousands of ads per request under strict latency (P99 below 100 ms), and the wide-beam, short-sequence shape leaves general attention kernels badly underused.
For eligibility, GRACE uses GTM (Generative Target Matching): at every decode step a candidate next token must satisfy two conditions, extending a valid SID prefix (catalog validity) and keeping at least one eligible ad under that prefix for this request. Low-cardinality attributes (country, age, gender) use bitmask matchers; high-cardinality attributes (fine-grained location) use Bloom filters. Because the stored matcher is an OR over all ads under a prefix, the test is conservative and can produce false positives, so an exact CPU-side ad-level check still runs after decoding.
For compute, GRACE rewrites the decoder. It targets a lightweight encoder-decoder (not an LLM), and its wide-beam, short-sequence shape is the whole point: in cross-attention all beams of one request share the same user-context KV, so GRACE treats beams as the query dimension and computes them at once (68 to 98× faster than FlashAttention-2/3); self-attention coalesces many beam rows into one large tile with a block-diagonal mask; a PagedAttention-style block table manages KV so beam rearrangement copies block ids instead of past KV; and beam size shifts from a fixed 1024 to a per-step dynamic schedule (1, 512, 1024, 1024), cutting 16.2% of FLOPs.
Evaluated on NVIDIA GH200 with 30M SIDs:
| Metric | Improvement |
| Ad-level targeting pass rate | 23.55% → 40.42% |
| Cross-attention latency | 68 to 98× faster |
| Self-attention latency | 23.4 to 25.8× faster |
| Decoder end-to-end latency | 197.7 ms → 17.8 ms (fixed beam); 15.8 ms dynamic |
With GTM (bitmask plus Bloom) added, full-model P99 is 53.6 ms, still inside the 70 ms compute window. k-way partitioning lowers matcher saturation but does not materially raise the final pass rate, so unpartitioned CD+GTM is the preferred operating point.
This paper pushes generative retrieval into production ads by attacking two real blockers: eligibility has to happen inside the decoding loop, and compute has to fit a brutal latency budget. For anyone doing generative-recommendation engineering, GTM is a concrete recipe for injecting request-dependent personalized constraints into constrained decoding, and the bitmask-plus-Bloom combination is reusable. The decoder work also shows that general LLM-serving kernels are badly mismatched to the wide-beam, short-sequence shape, where custom kernels buy one to two orders of magnitude.
Even with GTM the final ad-level pass rate is only 40.42%, meaning nearly six in ten generated ads are still ineligible and get discarded downstream; the waste is real. The location data is synthetic (64 random locations per user), which may not match real targeting distributions. All evaluation is on a single GH200 card and one encoder-decoder configuration, with no cross-hardware or cross-scale comparison. There is no explicit limitations section, and the still-low absolute pass rate is something the authors leave undiscussed.