Benchmarking and Enhancing Skill-Level Memory for Partially Observable Robotic Manipulation
Yansong Shi, Jiange Yang, Xijie Yang, Shaowei Zhang, Yuhan Zhu, Tao Lu, Limin Wang
cs.RO
2026-09-30
On HIDE, a 15-task benchmark where manipulation decisions depend on hidden state, robot foundation models all score below 27%; SEEK's three-memory design hits 62.9%, and 89% on a real robot.
Tell a robot to press the red button three times, and halfway through the current camera frame stops being enough: after the first press and after the second, the scene looks nearly identical, and whether to press again depends only on what already happened. The paper calls these decision variables hidden task states: information that cannot be recovered from the current visual and proprioceptive observation, only from interaction history. In POMDP terms, this is observation aliasing: two different histories produce visually equivalent observations that demand different actions.
Existing benchmarks don't isolate this. RLBench, CALVIN and LIBERO measure completion and generalization; recent memory-oriented benchmarks (RoboMME, RoboMemArena, RMBench) gauge memory demand mostly by trajectory length or scene complexity. Keeping history is also not the same as recovering the state a decision needs: on HIDE, SAM2Act+ scores 60.8% on progress tracking but 46.4% on repetition counting and historical recall, while MME gets 35.2% on recall and 2.4% on counting. The bottleneck is matching memory design to task information requirements, not memory capacity. The work comes from Shanghai AI Laboratory with USTC, Zhejiang University, Shanghai Jiao Tong University and Nanjing University.
HIDE extends RLBench with 15 tasks, five per category:
Demonstrations come from the scripted RLBench expert, and each trajectory is automatically segmented into stages at predefined keypoints, so structured stage labels arrive without manual annotation. Each task gets 100 training demos and 25 test episodes, and one policy is trained jointly across all 15 tasks. The design principle is explicit: long horizons plus occlusion don't make a task memory-dependent; tasks embed decision points where history is the only evidence.
SEEK adds three complementary memories to a SAM2Act backbone (a language-conditioned, coarse-to-fine multi-view policy):
Memory entries are written after action prediction, fusing visual features with the coarse branch's heatmap so spatial location survives; reading uses SAM 2-style memory attention with a bounded budget of at most 6 recent entries, one anchor and one stage state per view, independent of episode length. Training runs in two steps: first a memory-free policy with LoRA-adapted vision, then the visual backbone is frozen and only the memory modules train, separating learning to see from learning to use history.
Success rates on HIDE:
| Method | Avg | RC | HSR | EPT |
| OpenVLA | 1.6% | 3.2% | 1.6% | 0.0% |
| π0 | 22.1% | 10.4% | 36.0% | 20.0% |
| π0.5 | 14.4% | 5.6% | 24.8% | 12.8% |
| GR00T-N1.7 | 26.4% | 6.4% | 49.6% | 23.2% |
| RVT-2 | 42.9% | 43.2% | 42.4% | 43.2% |
| SAM2Act+ | 51.2% | 46.4% | 46.4% | 60.8% |
| SEEK | 62.9% | 61.6% | 59.2% | 68.0% |
SEEK leads the strongest baseline by 11.7 points overall, with per-category gains of 15.2, 9.6 and 7.2 over each category's best baseline. The VLA side collapses: the best of the five foundation models, GR00T-N1.7, reaches only 26.4%, and π0.5 lands at 14.4% with 5.6% on counting. These models can grab the button; they can't count their own presses.
Real robot (Franka Panda, 50 demos and 25 trials per task): SEEK averages 89% across four tasks, versus 47% for SAM2Act+ and 13% for π0.5.
| Real task | π0.5 | SAM2Act+ | SEEK |
| Press button ×N | 0% | 32% | 76% |
| Stack N cups | 20% | 48% | 100% |
| Clean desk | 0% | 44% | 80% |
| Find hidden chip | 32% | 64% | 100% |
Ablations back the design. Storing visual features instead of proprioception lifts the average from 42.5% to 62.9%, with the largest gain on counting (+30.4 points). Memory length from 0 to 6 moves the average from 46.7% to 62.9%; at length 2, counting actually drops from 46.4% to 44.2%, so a short window is not enough for that category. Single components each have a home turf: SCM on counting (+14.0), WCM on progress tracking (+12.0), PAM on recall (+11.6); individual mechanisms help some categories while degrading others, and the combination leads all three. Standard skills hold up: 84.7% on the 18-task RLBench versus 84.1% without memory, and 61.9% under The Colosseum perturbations versus 61.5%, with a slightly smaller relative drop (9.8% vs 10.1%).
Two things practitioners can take directly. HIDE works as a diagnostic: memory is decomposed into three operational requirements, so a category-level score drop shows which mechanism is missing instead of leaving you to guess after a real-robot failure. SEEK works as a recipe: the read budget is bounded (6+1+1 per view, regardless of episode length), it costs nothing on standard benchmarks, and adding memory doesn't trade away general manipulation skill.
The structural reading is blunt: foundation VLAs fail these tasks because nothing in the policy maintains internal execution state. An instruction specifies a goal, and "repeat twice" carries no information about how many are already done. For teams wanting to fix this, an explicit counter with boundary supervision is the cheapest starting point.
The authors state that HIDE covers only controlled, interpretable forms of hidden state; richer real-world partial observability and more general memory mechanisms are left to future work.
Open questions from the experiments: SCM's stage supervision comes free from scripted-keypoint segmentation, and real tasks don't ship stage labels, so how the supervision transfers is unanswered. At inference the counter advances on the policy's own boundary predictions, errors compound, and boundary-prediction accuracy is never reported, leaving it unclear how much of the residual failure is miscounting versus poor manipulation. VLA baseline adaptation is under-documented (official weights, reproductions and published numbers are mixed), so how much of π0.5's 14.4% is protocol disadvantage is unclear, and π0 outscoring π0.5 by 7.7 points goes unexplained. The real-world evaluation covers 4 tasks at 25 trials each, designed by the same team that built the method, which is close to grading your own exam.