A ladder of weaker feedback signals: ReAct, RLEF, Constitutional AI — and Math-Shepherd's rollouts
le_james94 · x · 2026-09-16
This post frames a 'ladder of weaker signals' in AI feedback loops: an environment answers (ReAct), a test suite answers (RLEF), a written document answers (Constitutional AI) — the loop never changes, but each source is cheaper to obtain and vaguer to act on. Above it: process reward models score every reasoning step (training one took 800K human labels), while Math-Shepherd swaps annotators for rollouts — a step is good if the model can still reach the right answer from it — lifting GSM8K from 77.9% to 84.1%.
Related event: Three Weeks with Stanford CS329A: Generators Have Outrun Verifiers(9 posts)→
More from Research
- GDPx: SemiAnalysis's frontier benchmark across the 11 most valuable domains — rohanpaul_ai · 2026-11-02
- Detecting covert instruction-following via internal representations parallels J-space work — dioscuri · 2026-11-02
- Xiaomi MiMo's CodeSpecWeave trains coding agents on interdependent specs for long-horizon coding — XiaomiMiMo · 2026-11-02
- Mistral Large 4 writes a quantum chemistry SCF solver from scratch with one short prompt — MistralAI · 2026-11-02
- Hunting for the most concrete "curing cancer with AI" roadmap of 2025 — iskander · 2026-11-02
- New theory paper proves linear decay is the unique minimax-optimal stepsize schedule for SGD — konstmish · 2026-11-02