ReuseGR decouples generative retrieval: 7B offline planner plus 0.5B online model cuts decoding cost
_reachsumit · x · 2026-11-05
ReuseGR splits generative retrieval into an offline/online pipeline: a 7B offline planner generates SID prefix sets shared across requests with the same query, while a 0.5B online model uses each user's history to pick prefixes and complete the remaining tokens — reusing large-model computation without losing personalization.
Key techniques:
- ReuseCut: adaptively picks cache depth per query based on historical coverage, trading off compute reuse vs. candidate coverage
- Path KD: path-level distillation transfers 7B suffix-scoring knowledge into the 0.5B online model
- BudgetCut: selects scored branches to improve top-100 recall under a fixed branch budget, initialized via on-policy distillation and refined with GRPO
Evaluated on an industrial search-advertising dataset with 2M query-item records and 3.38M catalog items.
More from Research
- Four generations of protein design models chart the road to fully programmable protein design — AllThingsApx · 2026-11-05
- ProteinMPNN likely memorized transmembrane folds; 0.3 noise breaks the recognition — sokrypton · 2026-11-05
- lm-eval Pitfall: Chatty Models Break Letter-Only Prompts, Regex Grabs Wrong Answers — mancubus77 · 2026-11-05
- Evolution can't copy homework: why species improve slower than iPhones — louisvarge · 2026-11-05
- Weaviate Podcast: Forgetting Is the Key to Scaling Agent Discovery, With Berkeley's Shangyin Tan — CShorten30 · 2026-11-05
- Epoch AI rebuilds its ECI with Bayesian methodology for sharper model comparisons — scaling01 · 2026-11-05