Multi-Budget Training Makes Retrieval Model Compression More General
antoine_chaffin · x · 2026-07-07
Instead of training for a single budget, the author trained the model with multiple compression budgets ([4,8,16,32,64,128,300]) at each training step. This multi-budget strategy makes the model easier to pool at any compression level. However, if the deployment target is known, training specifically for r=32 can retain 1.6% more information (reaching 97.4% compared to 95.8% for multi-budget) under the same λ, without sacrificing full token quality.
Related event: hpool Enables Flexible and Adaptive Compression for Retrieval Models(5 posts)→
More from Research
- RSNA 2026 Deep Learning Labs return: eight hands-on AI classes over four days in Chicago — PRouzrokh · 2026-10-27
- Paper: data schedules and learning rate schedules should be designed together — alifcoder · 2026-10-27
- Health benchmark study: quality and diversity of eval items beat quantity — keunwoochoi · 2026-10-27
- Across 7 tasks, highest reasoning setting isn't the best for most open models — keunwoochoi · 2026-10-27
- Encrypted GPT-2 speedrun challenges FHE inference: best is 3.031 s/token, goal sub-1s — tensorqt · 2026-10-27
- Open Models Match GPT-6 on MedMCQA but Trail 17 Points on Hard Clinical Cases — keunwoochoi · 2026-10-27