Hierarchical Self-Improvement: A Framework for Task-Specific Evolvable Agent Harnesses
Tailin Zhou
cs.AI
2026-08-09
HSI uses one frozen DeepSeek-V4-Flash to rewrite its own harness. BALROG BabyAI rises 42.0 to 81.3, Crafter 11.6 to 44.6; NLE stays near 0.
A large share of the spread in LLM agent scores comes from the harness: prompts, tool wiring, memory, and checkers. Swap the scaffold around the same model and the number can move by a wide margin. Two research lines have not closed this. Gödel-style systems such as Gödel Agent and the Darwin Gödel Machine mostly rewrite per-step decision code. Harness-engineering work such as Meta-Harness and Self-Harness often needs a stronger external proposer. A recent evaluation also found that automatic harness updates overfit and can lose to extra test-time reasoning.
The actual question is narrower. If the weights stay frozen, can an agent rewrite its own harness, and what stops that process?
Tailin Zhou at HKUST proposes Hierarchical Self-Improvement (HSI). One frozen DeepSeek-V4-Flash does three jobs, in three scopes with separate memory, sharing a single react() loop:
That lets the harness and the rewrite policy both move, without unbounded self-reference. Thinking is off during task execution and on during code edits, so later gains cannot be blamed on extra test-time reasoning.
Each run has 5 outer iterations and at most 80 react() steps. Seed selection picks an ancestor from the evolution graph and writes a hypothesis: which version, why, which direction, how to falsify it. Main evolution may change anything inside H except the task-injection seam usingharness, which stays fixed so versions can be hot-swapped and compared. Commit selection keeps a pool of branches, not only the top score. Final export picks by validation. Candidates are ranked with a noisy lower-confidence bound r = μ − 0.5σ/√n; reported numbers are raw % Progress. Evolution is not allowed to replace the model with external search or a non-LLM policy, but it may add tools and control logic inside the harness. On BabaIsAI-Make the trajectory still grew a BFS pathfinder.
Evaluation is on BALROG, six long-horizon text games. The controlled baseline is the original hand-written harness on the same DeepSeek-V4-Flash with thinking off at task time. Public leaderboard rows are context only; those models run their native reasoning settings.
Setup A uses the same task set with freshly sampled episode seeds:
| Env | Init harness | HSI meta-off | HSI meta-on |
| BabyAI | 42.0 | 77.3 | 81.3 |
| Crafter | 11.6 | 36.4 | 44.6 |
| TextWorld | 40.0 | 46.0 | 65.0 |
| MiniHack | 0.8 | 5.8 | 15.8 |
| NLE | 0.0 | 0.0 | 0.2 |
| Avg | 18.9 | 33.1 | 41.4 |
Against the init harness, meta-on is +39.3 on BabyAI, +33.0 on Crafter, +25.0 on TextWorld, +15.0 on MiniHack. Turning meta-evolution off drops TextWorld to 46.0 and MiniHack to 5.8. NLE barely moves.
Against the public board, TextWorld 65.0 sits above Grok-4 at 62.9 and Claude-Opus-4.5-Thinking at 59.0; Crafter 44.6 sits above GPT-5-minimal-think at 39.1. The unweighted average 41.4 still trails Gemini-3-Pro at 52.1. Reasoning budgets differ, so this is not a fair head-to-head.
Setup B holds out 20% of BabaIsAI sub-suites. BreakStop rises from 0.03 to 0.98, GoTo from 0.18 to 1.00. Make goes from 0 to 0.36 with large variance. On Crafter, dev reward climbs from 0.166 to 0.578 by iteration 4, then regresses in iteration 5. Evolution is not monotonic.
This is a weights-frozen, scaffold-only axis. For agents already in production that cannot touch the backbone, HSI is a concrete template: one harness per task family, hot-swapped through a fixed seam, selected by environment reward. Thinking-off at task time strips "think longer" out of the gain, so the numbers sit closer to the harness itself.
It will not yield one universal harness. BabaIsAI navigation transfers to unseen tasks; Make does not. The scaling story in the paper is continuous per-family evolution, not a single scaffold trained once. Code is public. It is a solo project with a limited compute budget and a single backbone.
The paper names two hard bounds. Feedback fidelity: sparse rewards give evolution nothing to select on. Backbone capability: harness edits cannot cover a task the frozen model cannot reach. NLE hits both.
The experiments also leave gaps. Setup A resamples seeds on the same task distribution; it is not a task-level holdout. Generalization is measured only on three BabaIsAI sub-suites. The whole study is one lineage, one population, one DeepSeek-V4-Flash. There is no matched ablation against the same model with thinking on, or against extra test-time search, even though a paper the authors cite warns that harness evolution can lose to test-time scaling. Appendix Table 3 marks TextWorld meta as off, while Table 1 reports meta-on 65.0; the configs do not match. The main text forbids replacing the model with external search, yet the Make trajectory grows BFS. The init harness is not pre-evaluated, and iteration 1 starts cold, so how the baseline was estimated is unclear.
Useful as a "freeze the model, evolve the scaffold" lab. Do not expect it to lift a weak model past its capability ceiling.