AutoIndex learns document-representation programs, lifting retrieval recall +8.4% on average
CShorten30 · x · 2026-09-14
UMass Amherst and Databricks Mosaic Research released "AutoIndex: Learning Representation Programs for Retrieval". Instead of tuning retrievers or rerankers, it treats how documents become indexed representations as an explicit optimization target:
- Searches over executable programs that slice, enrich, normalize, reweight, or reorganize documents before indexing
- Closed-loop optimization: agents diagnose retrieval failures, synthesize candidate programs, rebuild the index, and keep only updates that improve validation Recall@100
- Evaluated on CRUMB, a heterogeneous retrieval benchmark, with BM25 fixed; learned programs improve recall on all 8 tasks, averaging +8.4% Recall@100 and +8.3% nDCG@10, with peaks of +30.5% and +43.6%
- Takeaway: document representation should be an optimization target, not a fixed preprocessing choice
Paper, code, and arXiv preprint are available.
More from Research
- KAIST Proposes PLC-DPO to Correct Noisy Preference Labels via Posterior Margins — kaist-ai · 2026-09-14
- Tencent Hunyuan Open-Sources SAS: End-to-End Sparse Attention via Context Ranking — Tencent-Hunyuan · 2026-09-14
- NUS releases LIT to break vision-action shortcuts in robot foundation models — NationalUniversityofSingapore · 2026-09-14
- CUHK-SZ's COBRA-Skills: contextual bandit-guided skill evolution for LLM agents — CUHKSZ · 2026-09-14
- Code-graph retrieval cuts tokens 75-82% vs grep, but its benchmark skips accuracy — coding-os · 2026-09-14
- PhD student wants AI to analyze 169 ovarian slides after Nature Aging paper read 1,112 across 7 organs — anantm · 2026-09-14