Memory as Plans: World-Action Modeling with Memory-Grounded Planning
Sizhe Zhao, Haozhe Xie, Weiyu Zhao, Chenchu Zhang, Huan Wang, Chenyang Wang, Qinglin Liu, Shengping Zhang
cs.RO
2026-09-10
MaP-WAM plans from long-horizon memory, then executes a fixed-length language-visual plan: 83.3% on RMBench, 78% on a Franka, ~827ms latency that stays flat as history grows.
Most vision-language-action policies and world-action models treat control as Markovian: the current frame, or a short fixed window, is assumed to be enough. Many real manipulation tasks are not. A button gets covered, blocks swap places, a number flashes on a whiteboard once. The current image no longer contains the fact the robot still needs.
Memory designs sit on a bad axis. Language summaries stay compact and drop the visual details that matter, color, pose, object identity. Growing visual windows keep those details and then drag executor latency and GPU memory up with every extra frame. LingBot-VA, a causal world-action model that conditions on the full observation prefix, hits an out-of-memory failure around 1,700 frames. Coverage and speed are tied to the same knob.
MaP-WAM's bet is that long visual history does not belong in the executor at every control step. History is needed to decide the next segment-level goal and how the scene should evolve. The executor can follow a compact plan.
Memory is stored as completed-segment records: a language instruction plus 8 uniformly sampled frames. Planning has two stages:
The executor is a World-Action-Progress (WAP) model. A pretrained video DiT is extended with an action expert (1.02B) and a progress expert (207M) in a Mixture-of-Transformers. It jointly predicts future visuals, an action chunk, and a normalized progress sequence. How long a plan should run is unknown up front. When mean predicted progress crosses τ=0.95, the system replans. Recursive progress drifts, so plan-observation alignment calibrates it: the current observation is matched to nearby visual-plan frames and progress is snapped to the best match. On a segment switch, real execution frames are resampled into memory. Generated plan frames do not stay in the long-term context.
Executor context length is fixed. The plan prefix is KV-cached. Planning runs only at segment boundaries, so per-chunk latency is dominated by action generation.
On RMBench, 50 official demos per task and 100 evaluation rollouts:
| Method | Average success |
| Diffusion Policy | 5.8% |
| π0.5 | 10.4% |
| X-VLA | 9.8% |
| Mem-0 | 42.0% |
| LingBot-VA | 77.1% |
| MaP-WAM | 83.3% |
Swap T and Press Button both reach 96%. Observe and Pick Up, which requires matching temporally split observations across dozens of object types, moves from 9% for the strongest baseline to 19%. On Rearrange Blocks LingBot-VA is at 100% and MaP-WAM at 66%. The average hides a real loss on that task.
On a 7-DoF Franka Research 3, Find Button is 88% and Press Buttons is 68% over 50 trials each, 78.0% combined. Progress ablations on three hard RMBench tasks: replacing progress with a done/not-done classifier drops the average to 37.0%; removing the progress condition drops it to 54.0%; removing calibration drops it to 73.7%; the full model is 96.7%. Press Button is the stress test for the progress condition, 96% to 18% without it, because press and release look alike.
Per-chunk executor latency stays near 827ms. A full-history executor with KV caching is about 4× slower at 1,500 frames than at zero history, and exceeds 80 GB around 1,700 frames.
For long-horizon manipulation, this splits "what to remember" from "how to move." Memory lives on the planner. Executor context no longer scales with task duration. That is a usable systems tradeoff, not another larger window.
The cost is equally clear. The method depends on existing segment labels. WAP is trained per task; only the planners are multi-task. This is not a drop-in generalist VLA. 827ms per action chunk is slow for high-rate control, and the paper does not report wall-clock control frequency.
The authors flag two issues. Memory and progress currently ride the segment structure already present in the benchmarks; unsegmented demos would need automatic segment discovery. Plan-observation alignment uses a training-free matching metric that may need a learned similarity in visually busy scenes.
Rearrange Blocks is weaker than LingBot-VA, which keeps the full history on the executor, so taking history away from the executor is not free. Real-robot evaluation covers only two task families. Whether 8 sparse frames keep arbitrary fine-grained evidence is shown only on RMBench.