Score Distillation for Compact Retrievers
_reachsumit · x · 2026-07-14
Score Distillation for Compact Dense Retrievers
This research proposes a score-only distillation protocol: training a more compact dense retriever using only the teacher model's output score vectors, without needing access to the teacher's internal parameters or additional supervision.
According to the paper, this method recovers up to 50% of the performance gap between the base model and the teacher model, demonstrating that effective distillation is possible using only black-box scores.
More from Research
- Paper proposes a CRED taxonomy and benchmark to measure research-error detectors — soumitrashukla9 · 2026-07-22
- OpenAI says long-horizon models need safety and alignment checks across full action sequences — rhiever · 2026-07-22
- A Reddit user proposes a consistency LoRA to keep anime and game scenes visually stable — ThirdWorldBoy21 · 2026-07-22
- Graph workload 854.graph500 enters SPEC CPU 2026 as a new CPU benchmark — Prof_DavidBader · 2026-07-22
- BlackboxNLP 2026 is recruiting extra reviewers after a high submission volume — hanjie_chen · 2026-07-22
- AWS shows self-distilled reasoning can preserve math and coding skills during SFT — AWS ML Blog · 2026-07-22