PonderPounce: A Pretrained MLLM as an Episode Context Engine for Robot Control
Suhwan Choi, Jaeyoon Jung, Sungkyung Kim, Yunsung Lee, Youngjae Yu
cs.RO, cs.AI
2026-08-25
Ponder keeps episode history in a Qwen3.5-9B MLLM and asynchronously sends one cognition token to a π0.5 controller. On RoboMME at 1× data this reaches 60.83% versus 44.51% for FrameSamp+Modul; scaling the context engine from 0.8B to 9B adds 10.79 points on the same interface.
Robots often need facts that have left the current frame: a briefly highlighted cube, an occluded object, a demonstrated order. Humans score 90.50% on RoboMME. A current-observation π0.5 scores 17.93%, and past actions only raise that to 19.73%. Prior fixes add sampled frames, external banks, retrieval, or compression. Each is a purpose-built memory mechanism.
PonderPounce asks whether a pretrained MLLM's native causal context can be the episode memory, while the fast controller stays architecturally unchanged.
Two systems. Ponder (System 2) starts from Qwen3.5-9B and appends the instruction, optional demonstrations, and observations to native context. Each query writes K=1 continuous cognition carriers from hidden states, with no decoding or pooling. Subgoal text and demonstration reasoning stay inside System 2. Pounce (System 1) is 3.6B π0.5 on RoboMME and 3B GR00T N1.5 on RoboCasa-DC. It sees the current observation, proprioception, the newest cognition, and that state's sinusoidally encoded age.
Both pretrained parts are trained jointly end to end, with no bridge pretraining. Flow matching supervises actions. Where labels exist, cross-entropy supervises the transition gate, subgoals, and demonstration reasoning. RoboCasa-DC has no such labels, so only the action loss remains. Evaluation runs both models at 1 Hz and plays action chunks at 20 Hz. A resident KV cache yields 78 ms p50 cognition refresh; fused kernels bring Pounce to 25 ms p50.
RoboMME, 16 tasks, 50 episodes each:
| Method | Memory | 1× avg | 9× avg |
| π0.5 | none | 17.93 | - |
| MemER | keyframe retrieval | 42.38 | - |
| FrameSamp+Modul | frame tokens + modulator | 44.51 | 57.88 |
| PonderPounce | native MLLM context | 60.83 | 75.54 |
Permanence and Reference lead (62.83 / 72.17 at 1× versus 25.11 / 36.33 for FrameSamp). Imitation still lags: 33.67 versus 51.39 at 1×, 48.00 versus 63.00 at 9×. Under the same controller and interface, a 9B context engine scores 60.83% against 50.04% for 0.8B. Randomly initialized 9B training collapses to 0% success.
Supervision matters. Dropping demonstration-reasoning targets falls to 48.21%; dropping all LM-head grounding falls to 27.96%. A separately trained subgoal-text interface that updates the controller only at predicted transitions scores 59.96%, inside the 2.63 point spread across three eval runs. Holding cognition between predicted transitions, on the reported checkpoint, collapses success to 1.83%.
On RoboCasa-DC's five held-out cross-embodiment tasks the method scores 12.5%±0.9 against 11.6% for SeeTraceAct. Replacing cognition with a learned null state yields 8.6%; removing the demonstration yields 9.0%. Absolute success is low, and the paper does not claim broad superiority.
Episode memory can be a larger pretrained context model plus a thin asynchronous interface, not another robotics-specific store. System 2 capacity can grow while the fast path stays at 20 Hz. Groups already on π0.5 or GR00T can attach this as a memory layer without rebuilding the controller.
It is not universal. Frame reuse still wins on imitation-heavy tasks. Continuous cognition and subgoal text nearly tie, so the channel is not magic. What accumulates in context, and whether it is refreshed in time, is what pays.
RoboMME gates, subgoals, and demonstration reasoning come from simulator templates. Published baselines do not get the same text supervision, so the comparison mixes architecture and labels. Evaluation is two simulated benchmarks, one cognition carrier, no real robot. Pairing 9B context with a 3-3.6B controller raises train and serve cost. Latency is batch-1, not concurrent throughput, and context is capped at 16K tokens. The held-state intervention does not test a policy trained for sparse delivery. The staleness diagnostic is teacher-forced loss, not closed-loop success. Random init never converged, so pretraining is not isolated under matched optimization.