Snorkel's binary-gated GRPO lifts Qwen3.8-27B from 31.9% to 40.7% on OpenAI's GDPval

ajratner · x · 2026-11-21

Snorkel AI trained Qwen3.8-27B on 1,105 expert-built tasks from its GDPval+ series. After 102K rollouts, pass@1 on OpenAI's GDPval gold set rose from 31.9% to 40.7%, and model deliverables now beat human experts in 44.0% of LLM-judged head-to-head comparisons (vs 36.4% for the base model).

Key details:

Takeaway: expert data + an algorithm that learns from failures is what makes RL work on rare-success professional tasks.

Original post →

More from Research

Research channel →