Beyond the Current Scene: Event-Referential Grasping with Active View Selection
Hyunjoon Lee, Haebeom Jung, Eunsung Cha, Daeun Lee, Yu-Chiang Frank Wang, Jaesung Choe, Jaesik Park
www
cs.RO, cs.CV
2026-09-30
SNU and NVIDIA's BeyondCSe reads event history to resolve 'pick up the object moved last', then actively moves a wrist camera to reveal occluded targets: 77% vs 55% grasp success.
Telling a robot 'pick up the object you moved last' creates two problems. The target is defined by its role in a past event, so nothing in the current scene identifies it. And by the time the robot acts, the object may sit inside a basket or behind other things, invisible from the starting view. Current-scene grounding methods like LERF-TOGO and GraspMolmo, which map an instruction onto a specific object in view, have no signal to work with. Memory-aware systems recover identity but stop there: VoLo, a VLM orchestrator, reasons the correct target from history and passes a text description to π0.5 and Cosmos3-Nano-Policy, and in the paper's opening example both downstream models fail to grasp the occluded target. Knowing which object is meant does not tell the robot where to look now. BeyondCSe uses the event history for both decisions at once: what the target is, and where to move the camera.
The system is zero-shot, assembled from public components: Qwen3-VL-8B for all language-vision reasoning, CoTracker3 for point tracking, AnyGrasp for grasp poses.
Video reasoning runs four stages. RECORD watches the video without the instruction and writes a time-ordered record of (action, object, other object) triples. SELECT reads that record plus the instruction and produces an appearance description of the target and an event cue that distinguishes it. LOCATE turns the cue into a box in the initial image, and POINT clicks an action pixel inside the crop, which shrinks the search region and enlarges small parts like a cup handle. If the initial image yields no usable pixel, the system mines the history: it scans past frames from recent to early for pixels matching the target, tracks candidates with CoTracker3, rejects trajectories still visible at the end of the video, and lifts the surviving track samples into the robot base frame with depth, producing a 3D event track.
Occlusion goes to the active-perception module. The initial RGB-D frame builds a TSDF map, with regions unresolved by occlusion or missing depth forming the searchable workspace. The target belief starts as a Gaussian centered on the last track observation, its covariance shaped by the tail of the path, mixed with a 10% uniform component so the search can recover when the history-based estimate is wrong. Candidate views are scored by how much belief mass a negative observation would remove. One detail matters: confidence along a ray decays exponentially with the distance it travels through unobserved space, borrowing the transmittance form from NeRF without learning a radiance field, so 'saw nothing' is never read as 'nothing is there'. Each new frame triggers a Bayesian update whose likelihoods stay in (0,1]; a miss downweights locations instead of ruling them out. POINT re-queries the target at every new view until it lands a pixel or the view budget runs out.
All numbers come from a ROBOTIS OMY-F3M with a wrist-mounted RealSense D435i: 150 trials, 50 with visible targets and 100 with occluded ones. Baselines ran under original instructions and under queries reasoned by Qwen3-VL-8B from the history video.
| Setting | Best baseline | BeyondCSe |
| Initially visible, grasp success (50 trials) | 40% | 76% |
| Initially occluded, grasp success (100 trials) | 55% | 77% |
| All 150 trials | 50.0% (GraspMolmo) | 76.7% |
Localization succeeds on 43/50 visible and 88/100 occluded targets, 32 and 20 points ahead of the strongest baselines. Under original instructions the baselines collapse: LERF-TOGO and GraspSplats at 0% overall, GraspMolmo at 22.7%. A second comparison uses four heavily occluded scenes where targets are invisible even from a bird's-eye view, against an adaptation of Breyer et al.'s next-best-view method given the ground-truth 3D box:
| Method | Mean views | Grasp success (20 trials) |
| Breyer et al. (with GT box) | 3.35 | 75% |
| BeyondCSe | 2.20 | 95% |
Ablations isolate each part. Dropping the belief weights cuts success to 80% and inflates views to 4.65. Dropping the transmittance term raises views to 3.10. Dropping the event prior leaves success at 95% but nearly doubles views to 4.00. The prior buys search efficiency, not accuracy.
The paper wires memory and active perception into one loop. The lesson for anyone building embodied agents is concrete: a VLA or world model handed the correct target description still fails on occluded targets, and the missing step is the camera going to find its own evidence. The pipeline runs zero-shot on a single RTX 4090 from public components, so replication is cheap, and cutting the mean view count from 3.35 to 2.20 under heavy occlusion is real time saved on a physical robot. It is a systems paper, though: the components are known techniques, and the contribution sits in the problem definition and the volumetric belief-plus-transmittance formulation, at tabletop scale.
The authors state the main one: the target is assumed stationary during search, and objects that keep moving are future work. The qualitative section adds that LOCATE proposed an inaccurate region under large viewpoint changes in an EPIC-KITCHENS clip, with POINT recovering inside the expanded crop. Questions the paper leaves open: 30 scene-query pairs plus 4 heavy-occlusion scenes is a small evaluation; instructions are mostly event-order templates like 'moved first' or 'washed second', with free-form references untested; histories are short tabletop clips sampled at 96 frames, so there is no evidence on how RECORD degrades over long videos; and everything runs on one robot. Localization ground truth uses human-annotated 3D boxes with no distance margin, applied uniformly but sensitive to how tightly the boxes are drawn.