InterEvolve: Test-Time Evolution of Reward Programs for Humanoid Loco-Manipulation
Zhuo Lin, Sirui Xu, Liuyu Bian, Yu-Xiong Wang, Liang-Yan Gui
cs.RO, cs.CV, cs.GR
2026-10-02
An LLM rewrites reward programs from rollout feedback, teaching a frozen humanoid controller new tasks: 34.6%→86.5% success across eight families, deployed on a real Unitree G1.
Learned humanoid controllers handle contact-rich loco-manipulation as long as the task appeared in training. Outside that envelope they stall. A controller trained on human demonstrations of lifting, pushing, and carrying boxes has no demonstration for tipping a box onto another face or kicking it to a mark with its feet, even though the required motions sit inside its existing motor competence. The standard task interfaces all presuppose the answer: motion references, goal states, and skill labels ask for a desired motion, target state, or predefined behavior that is hard to design for nuanced multi-stage contact. Rewards are expressive enough for such interaction and show where an attempt fell short, but by convention each revision trains a new policy, which makes iterating on them slow.
InterEvolve keeps the controller frozen and moves task knowledge into reward programs that evolve at test time.
The execution layer is an object-aware FB behavioral foundation model. FB representations map any reward to a latent prompt that one fixed policy executes, so a new reward costs a new prompt instead of a training run. Existing humanoid FB models observe only the body, so two rewards that differ only in the object's goal collapse into one behavior. This paper attaches trainable object residuals to the frozen body branches of the actor, forward map, and backward map, and trains them on OMOMO and GRAB human-object interaction data retargeted to the Unitree G1. Execution runs through a fixed bank of about 50k states: a softmax-weighted, reward-tilted aggregate of backward embeddings forms the latent, recomputed at every control step.
A task is written as a reward program, an ordered list of stages, each with reward code and a completion condition, plus tunable constants. Two nested loops evolve it. An outer loop lets an LLM agent (DeepSeek-V4-Flash) revise program structure in context, guided by per-criterion rollout feedback and a skill library of verified programs; an inner loop tunes each proposal's constants with CMA-ES. Every candidate runs across 192 parallel simulation scenarios. A verifier is parsed once from the task text and then frozen, so the agent cannot rewrite its own exam. Feedback is criterion- and stage-resolved, e.g. 'only 93 of 192 environments reach stage 1', which points the agent at the stage to repair.
The two-loop split is deliberate. An LLM is good at choosing reward terms and stages; a numerical optimizer is better at finding the constants that make a given structure work. A promising structure still fails under poor weights, so neither loop suffices alone.
Reference tracking on held-out clips:
| Controller | Eh (cm) | Eo (cm) | SR |
| ULTRA (dedicated tracker) | 15.68 | 21.45 | 66% |
| BFM-Zero (body-only) | 28.36 | 62.33 | 8% |
| InterEvolve motor model | 27.20 | 30.80 | 60% |
| Motor model + evolved reward | 19.80 | 24.92 | 72% |
The agent discovered an object-anchored drift-correction reward that supplies the world-frame signal a locally observed actor cannot see, and success passes the dedicated tracker while tracking errors stay higher.
Goal-conditioned tasks, averaged over eight families (push to mark, carrying at two heights, tipping, feet-only interaction, lift to support, gates, obstacles, kicking):
| Program | Tuned | Structure evolving | SR | Earned |
| Human-written | no | no | 8.0% | 65.2% |
| Agent initial | no | no | 32.2% | 73.3% |
| Agent initial | yes | no | 34.6% | 82.5% |
| InterEvolve | yes | yes | 86.5% | 95.6% |
Standing still already earns 49.1% of tiers, so SR is the more discriminating metric. Calibration alone stalls at 34.6%; the more-than-doubling to 86.5% comes from structural edits. Ablations: forcing a single stage drops success to 44.7% and removing CMA-ES to 51.6%, the two largest effects. Round one delivers most of the gain (32.2% to 75.0%), with a dip at round three before the final 86.5%. On composite tasks, the full skill library solves 8/10 relocation, 4/10 stacking, and 5/10 carry-place-kick episodes, where no library solves almost none. Evolution costs 2.1 GPU-hours and 0.23M tokens per task family; per round, simulation takes 16 to 34 minutes on one GPU against 1 to 2 minutes of LLM calls. Programs evolved and verified in simulation ran autonomously on a physical Unitree G1 from an egocentric camera, with FoundationPose estimating the object's 6D pose.
This is a third answer to where task knowledge lives on a humanoid. Policy weights mean retraining per task; hand-designed references and skill lists mean per-task engineering. Reward programs that an LLM can read, edit, and test by execution drop the marginal cost of a new task from an RL training run to a batch of rollouts plus a few LLM calls. Controller and agent improve on separate timescales, and neither retrains the other. It is test-time scaling imported into robot control.
Kicking to a mark stays hardest because kicks are rare in the training data. Success is not monotonic across rounds; fixing one criterion can briefly break another. Simulation dominates cost, and the authors' proposed remedy is a faster simulator (mjlab), which concedes that iteration speed is the bottleneck. Real-robot results cover two tasks, both evolved in simulation first; nothing evolves in the physical world, and object state depends on pose estimation. The reward-tilted aggregation is acknowledged as a biased heuristic whose tilt and bank size need care: a 200k bank underperforms a 100k one. Composite-task success sits near half. No external baseline applies to this setting, so every comparison is a variation of the same system.