PUST: Transferring Update Signals via Proxy Models
KnowledgeXLab · hf · 2026-07-14
This paper introduces a novel post-training paradigm, **Proxy-guided Update Signal Transfer (PUST)**, aiming to decouple "exploration" from "distribution alignment". ### Core Idea - Instead of making the main model bear the expensive exploration costs directly, a lightweight **proxy model** is used to first search for high-reward behaviors. - The **relative improvement signal** before and after the proxy's optimization is extracted and transferred to the main model for policy alignment. ### Main Benefits - Reduces exploration costs and supports asynchronous generation, caching, and reuse of optimization signals. - It transfers "relative improvements" rather than absolute distributions, naturally supporting **weak-to-strong improvement** and cross-model transfer. ### Experimental Conclusions - Systematically evaluated on math and coding tasks using **Qwen3-family** models. - Results show that update signals from significantly weaker proxies can still consistently improve stronger main models with tunable performance gains. The authors reframe post-training from a "monolithic online optimization" process into a more modular, reusable, and cost-effective pipeline.
Related event: PUST: New Paradigm Uses Small Models to Guide Large Model Training(2 posts)→
More from Research
- GigaChat Audio targets long-form audio grounding with timestamps across 120-minute inputs — ai-sage · 2026-07-21
- Paper models Transformer components as stochastic geometry and tests five architectures — Zhihua Liang · 2026-07-21
- LTX 2.3 LoRA demo changes a video’s camera angle — CQDSN · 2026-07-21
- OpenForecaster uses daily news to improve language-model forecasting — Cohere_Labs · 2026-07-21
- Baseten study finds new facts in LLM weights are fragile unless trained from many restatements — alex_verem · 2026-07-21
- uv-scripts/ocr returns to the top of Hugging Face datasets with a JSON model picker — vanstriendaniel · 2026-07-21