Reflexion: Language Agents with Verbal Reinforcement Learning
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, Shunyu Yao
cs.AI, cs.CL, cs.LG
2023-03-21
Reflexion has agents write a verbal self-critique after each failure, store it in memory, and retry with lessons learned, with no weight updates, lifting GPT-4's HumanEval from 80% to 91%.
Getting language agents to learn from trial and error has meant reinforcement learning: reward signals, weight updates, many samples, expensive fine-tuning, and a policy that does not transfer. Reflexion asks a cheaper question. Since the model already writes natural language, can it write down where this attempt went wrong and what to change, store it, and read it back on the next try?
Reflexion splits the agent into three roles and runs a loop capped at roughly a dozen trials.
That critique goes into a bounded memory store (usually the last one to three entries), which the Actor reads on the next attempt. The key design choice is amplifying a scalar reward into readable, reusable verbal experience. Memory is short because the context window is short, and reflections have to be concrete to help.
Feedback sources are pragmatic. A 0/1 success flag, compiler errors, hand-written heuristics, or self-generated unit tests all work.
Code is where the gains show most. On HumanEval, GPT-4 alone scores 80%; with Reflexion it reaches 91%, beating the prior CodeT plus GPT-3.5 result of 65.8%. On LeetcodeHardGym, a set of hard problems post-dating GPT-4's cutoff, GPT-4 manages 7.5% and Reflexion doubles it to 15%. MBPP (Python) is the rare loss, 77.1% versus GPT-4's 80.1%. The paper explains this through the false-positive rate of auto-generated tests, 16.3% on MBPP versus 1.4% on HumanEval, so passing the tests does not mean passing the task.
| Task | Prior SOTA | GPT-4 | Reflexion |
| HumanEval (Python) | 65.8% | 80.1% | 91.0% |
| HumanEval (Rust) | — | 60.0% | 68.0% |
| MBPP (Python) | 67.7% | 80.1% | 77.1% |
| Leetcode Hard (Python) | — | 7.5% | 15.0% |
On the decision task ALFWorld, Reflexion solves 130 of 134 tasks, a roughly 22-point absolute gain over twelve trials while the baseline's hallucination rate is stuck at 22%. On HotPotQA reasoning, CoT with ground-truth context rises from 60% to 74%. Ablations show both components are necessary: without test generation the agent cannot tell if it is right, and without reflection it cannot fix errors even when it spots them.
This is the citation you cannot avoid on how agents learn from failure. It moves learning from updating weights to updating the prompt context: cheap, interpretable, and open to anyone with an API key. Nearly every modern "let the model retry and write down what it learned" workflow descends from it. For practitioners it is a ready-made failure-retry template of evaluate, reflect, retry with memory.
The authors concede several. Performance depends on a strong base model: on the weaker StarChat-beta, Reflexion barely moves the needle, suggesting self-correction is an emergent property of large models. Memory is bounded and the context window limits how long or how many reflections you can carry. Reflection quality is not guaranteed, since the model can write the wrong cause into memory and reinforce its own error, which the paper does not test systematically. The evaluator can also be wrong, as the MBPP false positives show: garbage in, garbage out.