RL + Distillation Closes Over Half the Gap Between a 4B and 27B Model
Prompt Engineering · youtube · 2026-10-17
A Prompt Engineering video tests whether open-source RL environments can shrink the gap between small and large models. The author trains Qwen 3.5 4B with GRPO + LoRA on two DGX Sparks using Kaggle-style tasks, benchmarked against Qwen 27B on a held-out exam.
- Explains RL environment anatomy (sandbox, tools, checker) and why a strict checker is needed to avoid reward hacking; GRPO learns only from tasks the model sometimes solves
- Random tasks yield weak gains; filtering for "edge" tasks closes over half the gap, mostly by teaching the model to finish and answer correctly
- Progress stalls after further RL, so distillation from 27B followed by RL on top gives the best results on unseen datasets
- Code, checker, and recipes released on GitHub
More from Research
- Natural stimuli make V1 responses dirty via population effects — HistedLab · 2026-10-17
- Neuroscientist: strong polysemanticity found in nearly every brain area — HistedLab · 2026-10-17
- Google.org picks 17 organizations for $36M AI for Science Impact Challenge — ymatias · 2026-10-17
- Romik's ambidextrous sofa optimality formalized and proven in Lean 4 — IgorCarron · 2026-10-17
- AstraZeneca strikes $40 million AI drug discovery deal with insitro — HealthcareLdr · 2026-10-17
- YC Paper Club: Stanford AI Safety Directors on OpenAI-Hugging Face Incident and an NTSB for AI — Y Combinator · 2026-10-17