Hybrid product search straight from the data lake: Pinot + BM25 + RRF hits 6.5-9ms p50 on 277M products
_reachsumit · x · 2026-11-11
An arXiv paper shows e-commerce hybrid search can run directly on a data lake, eliminating separate lexical-engine and vector-DB catalog copies.
- Lakehouse table (Apache Iceberg over Parquet) as single system of record; Apache Pinot extended with BM25 scoring on its Lucene index, corpus-wide term statistics for comparable scores, and reciprocal rank fusion in plain SQL
- On Amazon ESCI (1.2M products, 22,458 test queries), lake-served hybrid search matches exact retrieval at nDCG@10 .361; tables queryable 149s after registration, lake commits searchable in 3.8s
- On a 277M-product (0.8TB) brand-sorted lake, one 16-vCPU node serves brand-scoped vector/BM25/hybrid searches at 6.5-9ms p50, sustaining 1,100 vector and 635 BM25 searches/sec under 180ms p99
One ingestion pipeline instead of three.
More from Infra
- Anthropic says Claude ported kernels to run on AMD GPUs in just 3 days — scottleibrand · 2026-11-11
- Chick-fil-A runs 2,800 edge Kubernetes clusters with billions of monthly MQTT messages — SumitGup · 2026-11-11
- RX 6800 gets 3x faster ComfyUI images by re-enabling silently disabled INT8 — TheCat001 · 2026-11-11
- Shopify's RoutingTrie lifts generative retrieval Hit@100 from 67.5% to 96.1% with 1.1ms overhead — _reachsumit · 2026-11-11
- CoreWeave revenue jumps 168% to $3.66B but net loss balloons 6x to $709M on borrowing costs — rohanpaul_ai · 2026-11-11
- NVIDIA open-sources NVKE and NVKR for managed Kubernetes on GPU infrastructure — HowDevelop · 2026-11-11