Snorkel's binary-gated GRPO lifts Qwen3.8-27B from 31.9% to 40.7% on OpenAI's GDPval
ajratner · x · 2026-11-21
Snorkel AI trained Qwen3.8-27B on 1,105 expert-built tasks from its GDPval+ series. After 102K rollouts, pass@1 on OpenAI's GDPval gold set rose from 31.9% to 40.7%, and model deliverables now beat human experts in 44.0% of LLM-judged head-to-head comparisons (vs 36.4% for the base model).
Key details:
- GDPval tasks come from experienced professionals across nine US economic sectors; each requires a real deliverable (financial model, regulatory filing, slide deck), averaging 50K response tokens and many tool calls per attempt. A pass means satisfying ≥90% of the rubric.
- The gains come from combining expert-built training data with binary-gated GRPO, their algorithm for learning from failure.
- Standard GRPO with a binary reward signal barely moves off the base model on the same data and rollout budget.
Takeaway: expert data + an algorithm that learns from failures is what makes RL work on rare-success professional tasks.
More from Research
- Artificial Analysis Launches Search API Benchmark Covering 26 Products Across 14 Providers — ArtificialAnlys · 2026-11-21
- The 250,000-page Riemann proof claim: when computational verification outpaces human comprehension — SydSteyerhart · 2026-11-21
- Genesis Molecular AI launches Ruby potency model, up to 1000x faster, open-sources training data — edunov · 2026-11-21
- 'All AI can do is plagiarize': model caught regurgitating a proof straight from The Book — moyix · 2026-11-21
- Experts gave AI 10% odds of cracking a Millennium Prize Problem by 2027 — two now claimed — theimposingshadow · 2026-11-21
- Dario says Anthropic found a new gene-editing mechanism; critics cite Feng Zhang's leaner record — max_paperclips · 2026-11-21