Small Models Explore First to Guide Large Models
burny_tech · x · 2026-07-15
This paper proposes PUST (Proxy Exploration and Reusable Guidance): traditional post-training often requires large models to perform expensive RL exploration themselves, whereas PUST shifts the exploration process to a smaller proxy model.
Core Idea
- The small model completes an exploration phase first, learning useful update directions
- When transferring to the large model, what's passed is the "update direction," not the proxy's final distribution
- This allows the small model to search and cache signals first, which are then reused for the larger model
Results
- The method brings improvements on math and coding tasks
- The paper mentions that signals generated using Qwen3 4B can significantly boost the performance of Qwen3 8B
Related event: PUST: New Paradigm Uses Small Models to Guide Large Model Training(2 posts)→
More from Research
- Google DeepMind pilots world's first double-blind evaluations for frontier models — scychan_brains · 2026-10-26
- Rewriting scaling law study: optimal synthetic data share rises from 10% to 32% with compute — heghbalz · 2026-10-26
- FLASH: GPU-native deformable sim learns cloth folding in minutes on one RTX 5090 — ChongZzZhang · 2026-10-26
- LEMON: zero-shot LLM misbehavior monitor flags hallucinations from attention maps — yoavartzi · 2026-10-26
- Behind the EMNLP keynote: LMs as simulated learners to generate child language acquisition hypotheses — najoungkim · 2026-10-26
- The paper behind the EMNLP 2026 keynote on bigram 'simulated learners' — najoungkim · 2026-10-26