Self-Refine: Iterative Refinement with Self-Feedback
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, Peter Clark
cs.CL, cs.AI, cs.LG
2023-03-31
Self-Refine loops one LLM through generate→feedback→refine with no training, gaining ~20 points absolute across 7 tasks; math reasoning barely moves (GPT-4: 92.9→93.1).
A single pass from an LLM is rarely its best output: dialogue replies come out bland, code runs slow, constrained generation misses keywords. The existing fix was task-specific refinement, either training a corrector per domain (PEER, Self-Correction) or paying for human annotations and reward models, all of which has to be redone for every new task. Self-Refine poses the cheaper question: with zero training, can a model serve as its own reviewer and editor?
The whole loop needs one LLM and three few-shot prompts: initial generation, feedback, and refinement.
Two choices carry the weight. Feedback is natural language rather than a scalar reward, because a pretrained model can already write prose critiques with no extra training. And the refine prompt carries the full history of drafts and feedback, so the model avoids repeating earlier mistakes. For multi-aspect tasks (an acronym should be pronounceable, spellable, and relevant), the feedback emits numeric scores per aspect and the final output is the draft with the best total, smoothing quality swings across rounds.
| Task (base → Self-Refine) | GPT-3.5 | ChatGPT | GPT-4 |
| Dialogue response (pref. rate) | 36.4→63.6 | 40.1→59.9 | 25.4→74.6 |
| Sentiment reversal (pref. rate) | 8.8→30.4 | 11.4→43.2 | 3.8→36.2 |
| Code optimization (% optimized) | 14.8→23.0 | 23.9→27.5 | 27.3→36.0 |
| Math reasoning (GSM8K solve rate) | 64.1→64.1 | 74.8→75.0 | 92.9→93.1 |
The baseline is one-shot generation from the same model; the average absolute gain is about 20 points. Against outside systems: on GSM8K with the same GPT-3 base, the trained Self-Correction corrector reaches 45.9% where Self-Refine gets 55.7%; on the PIE code-optimization set, GPT-4 with Self-Refine hits 36.0% using at most 4 samples, close to the 38.3% ceiling of few-shot PIE sampling 32 outputs (human rewrites: 38.2%).
Math barely moves, and the paper is blunt about why: the model cannot spot its own errors. ChatGPT's feedback on 94% of instances amounted to "everything looks good." Swap in an oracle that reports right or wrong, and GPT-3.5 immediately gains 4.8 points.
Ablations isolate the mechanism. Replacing specific feedback with generic advice ("improve the efficiency") drops sentiment reversal from 43.2 to 31.2, and removing feedback entirely zeroes the task. Gains diminish with iterations: code optimization climbs 22.0 → 28.8 over three rounds, most of it in round one. Against simply sampling 4 outputs and picking among them, human raters still prefer the Self-Refine result. In the 70-case failure analysis, 94% of failures trace back to bad feedback (mislocating the error or prescribing the wrong fix); only 6% come from the refiner botching good feedback.
This is one of the source works for test-time scaling: no new parameters, no training, just more inference compute. For practitioners it is a near-zero-cost rewriting loop that fits preference-driven, constraint-heavy tasks best. The math failure is just as informative: where correctness is objective but the model cannot self-check, pure self-feedback stalls, which is exactly the gap verifiers and process reward models later filled.
Stated by the authors: the base model must be strong (Vicuna-13B could not hold the feedback format and ignored even hardcoded correct feedback), all experiments ran on closed API models, and evaluation is English-only. Costs the paper leaves unreported: each output takes multiple extra model calls, and no token bill appears. Part of the scoring uses GPT-4 as a judge (68%–82% agreement with humans), and the human A/B study was run blind but by the authors themselves, 150 examples per task.