Qwen3-0.6B hits strong MS MARCO results via data curation and Model Souping
_reachsumit · x · 2026-10-14
A single-author report shows lightweight Qwen3-0.6B rankers achieve competitive passage ranking on MS MARCO without architectural changes, custom losses, or cross-encoder distillation.
- Gains come from data-centric strategies: corpus-targeted continued pre-training, rigorous curation, hard-negative mining, and train-inference distribution alignment
- Model Soup weight merging delivers marked gains while preserving single-model inference efficiency
- Validated across three settings: BM25-then-rerank, corpus-wide dense retrieval, and dense retrieval-then-rerank
- Shows sub-billion-parameter models can approach SOTA through data engineering alone
More from Research
- AIPOCH's open-source Open-Science workbench: reproducible AI research sessions — ahuja_priyank · 2026-10-14
- AIPOCH's Open-Science ships portable .science packages that share full research context, not just outputs — ahuja_priyank · 2026-10-14
- AIPOCH's Open-Science packages the full path to AI-assisted research results into .science files — ahuja_priyank · 2026-10-14
- Video-Arena: Game-Based Video Benchmark Shows Native Models Score Just 13.2 F1 — Kuan Zhang · 2026-10-14
- LSCBench: Vibe Coding Agents Write Functional Code, But Only 25% Is Also Secure — Qwen · 2026-10-14
- Omni-LLMs Know More Than They Say: NAVER's OmniCatalyst Adds 7.29% via Self-Generated Text — naver-ai · 2026-10-14