Dream-RSI: Recursive Self-Improvement through Evolving Worlds
Tong Zheng, Xidong Wu, Zheng Zhang, Zhankui He, Chaoyi Zhang, Benjamin Coleman, Ruoqiao Wei, Di Bai, Haolin Liu, Rui Liu, Xue Wang, Yue Zhuan, Wang-Cheng Kang, Renkai Xiang, Heng Huang, Xinwu Cheng, Yunsong Guo
cs.CL
2026-09-14
Dream-RSI replays past discovery trees as zero-cost simulators, matching Lasso, math and kernel quality at far lower agent-call cost.
Once coding agents can propose candidates, the scarce resource is exploration: which branch to grow, how much parallelism, when to stop. Hand-written strategies stay frozen as the search space grows. Online meta-optimization is expensive, because judging an exploration policy means watching a long proposal-eval rollout, and the space of policies is huge.
Dream-RSI treats a finished discovery tree as a replay simulator over the realized search space. A new policy is just a different walk: different branches, order, batching, and stopping. Node outcomes are already stored, so the walk does not call the coding agent again.
A thin orchestration layer makes those decisions programmable and leaves the discovery agent unchanged. Only the exploration-policy code is rewritten. The evaluator and execution interfaces stay fixed.
The loop has three stages. Online, the current policy grows a discovery tree and appends it to history. Offline, those trees become a simulator pool. A policy-development agent proposes candidate controllers and “dreams” them on recorded trees: from the root, each action reveals stored children. The replay score is best quality minus a cost on the number of revealed attempts, plus a bonus for attempts per decision round. After M revisions, the version with the best average replay score is deployed for the next online round, which grows a new tree and enlarges the pool.
Because the incumbent is in the candidate set, average replay score on the frozen history does not get worse. That guarantee is about replay, not about the next live tree.
The controlled baseline is Recursive Fixed Exploration: same agent, same initial parallel-refine policy, no later policy updates. On Lasso path discovery with Gemini-3.1-Pro, mean runtime on six held-out datasets falls from 3587.1 ms to 2931.0 ms, with 317 discovery-agent calls instead of 550. Gemini-3.7-Flash goes from 2516.7 ms / 3200 calls to 2350.6 ms / 1879. sklearn averages 44180.3 ms, glmnet 13767.5 ms. SimpleTES reports 3804.8 ms after 51,200 generations on GPT-OSS-120B, so 317 vs 51,200 is about 162× fewer calls across systems.
On three math tasks with Gemini-3.1-Pro and under 1,000 generations, Sum-Difference is 1.145427 versus SimpleTES 1.143975 and fixed exploration 1.144047. Circle Packing matches the strongest reported 2.635983. Autocorrelation is 1.456375, a bit behind SimpleTES at 1.453675 (lower is better).
On four KernelBench kernels versus the same Gemini-3.1-Pro fixed policy, VGG16 and LayerNorm reach similar speed with 2.43× and 1.79× fewer generations; ConvDiv and ConvMax score 2.09× and 1.44× higher under similar budgets. Injecting history as semantic prompt guidance hurts ConvDiv. The learned controller also adapts effort: evaluated attempts on ConvDiv drop from 110 to 50, then rise again after a plateau.
This is Dreamer-style policy improvement at the exploration layer, with a world that is not a learned dynamics model but a log of executed attempts. For AlphaEvolve-like loops, the practical move is to make exploration an executable policy and use past trees as a free meta-evaluator, instead of rerunning thousands of generations after every controller tweak.
The fair comparison is fixed exploration on the same backbone. The 162× figure versus SimpleTES mixes model families and should be read as an order of magnitude, not a clean speedup.
Replay cannot invent branches that were never grown; new directions still need live rollouts. The Sum-Difference gain is tiny in absolute terms, and Circle Packing’s shared 2.635983 looks like a known construction ceiling. Autocorrelation still loses to a 51,200-generation SimpleTES run. Policy edits come from another LLM rewriting code; the paper shows replay score does not drop, not that the improvement operator converges. There is no dedicated limitations section, and no measurement of how often the replay-best policy is actually best on the next live tree.