AutoIndex lifts CRUMB recall by 8.4% without changing retrievers or embeddings
mrdrozdov · x · 2026-07-22
AutoIndex does not settle on a single universal preprocessing recipe. Instead, it learns corpus-specific programs that can slice, normalize, enrich, reweight, and reorganize documents.
The point is not to find the best chunk size. It is to learn the representation program that exposes the most useful evidence to the retriever. To isolate that effect, the rest of the retrieval stack is kept fixed: BM25 retriever, ranking function, and indexing backend remain unchanged. Only the program that maps documents into indexed units is learned.
On CRUMB, that setup yields:
- Recall@100: +8.4%
- nDCG@10: +8.3%
The authors highlight that this is a meaningful gain without retriever fine-tuning, embedding updates, or online feedback.
More from coding & agent
- BasicallyMythos logs Claude Code reroutes and can swap the fallback model to Kimi K3 — JoshuaJBouw · 2026-07-22
- A Codex meme turns a flat “great” into the whole joke — Aizkmusic · 2026-07-22
- Auto-Company runs 14 role-play agents as a 24/7 autonomous company on your own machine — aigclink · 2026-07-22
- DSPy + RLM agent framework runs on Qwen-4B and keeps large context outside the model — dosco · 2026-07-22
- Fractal runs entirely on git worktrees, tmux, and a local SQLite database — rohanpaul_ai · 2026-07-22
- Fractal adds root budgets, Markdown memory files, and hard caps for recursive agents — rohanpaul_ai · 2026-07-22