RecoGEM: a production serving system for generative recommendation, 1,000+ QPS in deployment
_reachsumit · x · 2026-11-11
A new paper presents RecoGEM, a production serving system for generative recommendation (GR), where recommenders autoregressively generate item IDs instead of scoring candidates — promising better quality but far harder serving. RecoGEM decouples runtime from engine: candidate-state transitions and live artifact updates move to a dedicated runtime, while the execution engine exploits GR's short token sequences, huge candidate populations and predictable step shapes to optimize attention, candidate selection and execution planning. It has been deployed for over nine months, serving apps with hundreds of millions of users across 10+ GR model families, exceeding 1,000 QPS per instance on OneModel production workloads.
More from Infra
- PufferLib: open-source RL library trains 60M samples/sec on one GPU — jsuarez · 2026-11-11
- Rise of ASICs may erode NVIDIA's CUDA moat, commentator argues — bookwormengr · 2026-11-11
- Fable 5.2 tops SemiAnalysis' GDPx chip design leaderboard at 16.4%, but some tasks saw no gains — ycombinator · 2026-11-11
- Tencent's newest flagship Hy4 goes live, co-served by Inferact with open-weight preview — VoidAsuka · 2026-11-11
- Claude ported itself to AMD MI355X in ~3 days, work that took dozens of engineers 6-12 months — AnushElangovan · 2026-11-11
- Analyst: AI datacenter demand will eventually be built in space — TinfoilTricorn · 2026-11-11