Microsoft Research: representation-based exploration boosts verifier efficiency 50%+ and 3x sample efficiency in LLM RL
burny_tech · x · 2026-11-10
A paper by Jens Tuyls, Dylan J. Foster, Akshay Krishnamurthy and Jordan T. Ash (Microsoft Research), accepted at ICLR 2026, asks whether RL on LLMs discovers genuinely novel behaviors or merely sharpens existing ones.
Key finding: a simple, principled diversity bonus computed from the pre-trained model's hidden states substantially improves solution diversity and pass@k — in both inference-time scaling and RL post-training.
- Inference time (RepExp): iteratively select responses maximizing an elliptical representation bonus; Qwen-2.5-14B-Instruct gains over 50% verifier efficiency on nearly all tasks.
- Post-training: on AIME 2024, the post-trained Qwen-2.5-7B-Instruct matches GRPO's pass@256 with only pass@80 — a 3x sample-efficiency improvement over standard RL.
The authors conclude deliberate exploration with the right notion of diversity is a practical path to discovery beyond sharpening. Code and website are released.
More from Research
- New cryo-EM structure: pSTAT1 bound to importin alpha 5 (PDB 11GJ) — chaitjo · 2026-11-10
- ARLoop: fully-open 1.2B looped transformer beats Huginn-3.5B and Qwen3.5-2B on all 8 benchmarks — hhsun1 · 2026-11-10
- RAGCap-Bench: A Fine-Grained Benchmark for Intermediate Capabilities in Agentic RAG — kalyan_kpl · 2026-11-10
- RBA gives LLaVA-OneVision-2 a 3x lossless training speed-up, merged upstream — liuziwei7 · 2026-11-10
- Networks aren't scale invariant — so why has scale invariance motivated so many useful methods? — jasondeanlee · 2026-11-10
- SPACE-Tag preprint: high-throughput spatial chromatin profiling via CUT&Tag — anshulkundaje · 2026-11-10