AutoIndex learns document-representation programs, lifting retrieval recall +8.4% on average
CShorten30 · x · 2026-09-14
UMass Amherst and Databricks Mosaic Research released "AutoIndex: Learning Representation Programs for Retrieval". Instead of tuning retrievers or rerankers, it treats how documents become indexed representations as an explicit optimization target:
- Searches over executable programs that slice, enrich, normalize, reweight, or reorganize documents before indexing
- Closed-loop optimization: agents diagnose retrieval failures, synthesize candidate programs, rebuild the index, and keep only updates that improve validation Recall@100
- Evaluated on CRUMB, a heterogeneous retrieval benchmark, with BM25 fixed; learned programs improve recall on all 8 tasks, averaging +8.4% Recall@100 and +8.3% nDCG@10, with peaks of +30.5% and +43.6%
- Takeaway: document representation should be an optimization target, not a fixed preprocessing choice
Paper, code, and arXiv preprint are available.
Related event: AutoIndex: LLM-Written Code Optimizes Retrieval Indexing(3 posts)→
More from Research
- Cherimoya Binding Atlas BATv1: 46k-param genomic models for 3,429 TF binding experiments — jmschreiber91 · 2026-10-31
- Heads Of Security: paper pinpoints and rewires attention heads behind prompt injections — jonasgeiping · 2026-10-31
- BATv1: 17,145 Cherimoya models map TF binding across 3,429 ENCODE ChIP-seq experiments — jmschreiber91 · 2026-10-31
- UCLA's ChairNav teaches autonomous wheelchairs to learn each rider's preferences — zhoubolei · 2026-10-31
- How much compute should data rewriting get? New scaling-law study quantifies it — ChengleiSi · 2026-10-31
- Open-source sticker library covers all 11,064 living bird species, drawn by AI — AlchainHust · 2026-10-31