Do Agentic Policies Need to Look at Every Step? Geometric Abstraction for Reasoning and Action
Yihang Liu, Yiming Chen, Mingzhe Zheng
CoRL 2026 Workshop on Agenti
cs.RO, cs.AI, cs.CV
2026-11-03
Grounded 3D part symbols and text checkpoints replace per-step image reads. Online bridge completion rises from 24% to 56%; ring placement finishes 33.3% faster.
Multimodal models can already look at a robot scene and drive a manipulation step. The bill comes due in the loop. Once the scene changes, the model rereads the raw images, even though the decision often depends on a handful of geometric quantities: which face is down, whether two faces meet, and how far the part still sits from the goal. Recovering metric structure from pixels is unreliable, and most of the visual tokens are redundant.
A raw image keeps the whole scene and demands a fresh reading every step. A perception API that returns a center or a box is too coarse for mating faces. Keypoint constraints and executable programs each cover one side of the stack. Neither supplies a scene state whose symbols survive across decisions while the metric values are rewritten from the current view.
The geometric abstraction records the scene as typed 3D parts: centers, vertices, edges, and oriented faces, not a single centroid and not a vague caption. Every part has a timestamp and an evidence status. It was seen in the current depth, filled from a shape prior stored at initialization, or marked unavailable under occlusion.
Calibrated side and wrist RGB-D, together with proprioception, are back-projected into the world frame to refresh pose and size. Shape may stay. Pose may not. After an object moves, pose is estimated again from the new observation. It is not copied from the previous state, and it is not inferred from the command that was sent. If the estimate fails, the reference is unavailable, the goal is kept, and the residual is not computed from a stale pose.
Symbolic references name those parts. red.bottomface and blue.topface bind to coordinates only when the program runs, like a hole number on a drawing that is measured on site. For placement, the policy reads the two mating centers and outward normals and requests a rigid transform that sets the source normal against the target normal and matches the centers. Twist about the normal is still free, so align returns the smallest rotation and reports the leftover twist separately. A support check looks at relative height, planar overlap, and face orientation, then answers true, false, or unknown. True means the geometry looks seated. It does not certify contact or stability.
After each action, a textual checkpoint returns the relevant geometry, the evidence status, and the metric residual to the goal. The symbolic task structure stays. The numbers are filled in for this frame. Images can still be requested when the abstraction is not enough.
In policy synthesis, the language model edits an executable program only between reset rollouts. Grounding and control then run locally, with no further model calls. In online execution the scene is not reset. The model reads the new checkpoint, picks a skill, and recovers from the geometry observed after the action.
The simulator is an SO-100 arm with calibrated side and wrist RGB-D. Pixel Reasoning reads images and writes its own perception inside the generated program. Geometric Abstraction also receives sensor-grounded part references and checkpoints. Both sides get the same task text, actuators, and initial skills, and neither sees simulator ground-truth poses. Synthesis uses gpt-6-astra at high reasoning effort. Online execution uses the same model at medium effort.
Synthesis starts from an observation-only scaffold: five paired chains, five shared training layouts, three revisions. Each revision runs every layout, with a reset between rollouts and no model calls during execution. A candidate is kept only if it still solves every training layout already solved and adds at least one more. Otherwise the incumbent remains. After training freezes, each task is scored on 100 shared held-out layouts. Tokens are input plus output, include rejected proposals, and are averaged over the five chains.
| Comparison | Geometric Abstraction | Pixel Reasoning |
| Bridge + cylinder, R1 held-out SR | 45.2% | 21.2% |
| Three-cube stacking, R1 | 57.2% | 17.2% |
| Those two tasks, R2 | 63.4% / 72.2% | not reported |
| Those two tasks, R3 | 65.4% / 81.4% | 56.0% / 70.8% |
| R2 tokens, bridge / stacking | 67.3k / 64.7k | 125.7k / 138.1k |
| Ring-on-peg, grasp and thread, R3 | 38.0% | 0.0% |
At R2, 63.4% and 72.2% already pass Pixel Reasoning's final 56.0% and 70.8%, at about half the tokens. Ring-on-peg is 0% for both sides through R2. Only Geometric Abstraction reaches 38.0% at R3. The gain mixes program synthesis with local geometric grounding during execution. Inference speed and text formatting were not measured apart. By the last revision, bridge tokens are about even, stacking tokens are lower, and chain-to-chain variance is lower.
Online evaluation is 300 episodes, 50 paired layouts per task. Failures do not reset the scene. Model wait and motion share 200 seconds, and physics keeps running while the model waits. In every pair below, Geometric Abstraction comes first.
| Task | Completion | Time and cost |
| Build a bridge, then stack two cubes | 28/50 vs 12/50 | model time 6.7% lower, tokens 15.1% higher |
| Move the top two cubes to a new base | 42/50 vs 15/50 | model time 60.5% lower, tokens 52.5% lower; motion 104.5 s vs 103.2 s |
| Seat a pre-lifted ring on a peg | 50/50 vs 49/50 | 77.2 s vs 115.8 s, 33.3% faster, earlier on 45 of 50 pairs |
On the pre-lifted ring, request time falls from 90.0 s to 45.6 s, tokens from 214.5k to 101.3k, completed requests from 11.88 to 6.90, and attached images from 10.94 to 2.72. On bridge-then-stack, completion rises and token use rises with it.
A 200-second budget counts waiting as part of the task, so fewer image round-trips decide whether the episode finishes. Both sides already seat the pre-lifted ring. The difference there is time and attachments, 10.94 images down to 2.72. Construction is where completion moves: 24% to 56% for stacking on a bridge, 30% to 84% for moving the top two cubes.
The change is the scene interface the language model reads. gpt-6-astra still writes or selects the program, and object shape is given at initialization. If the local geometry holds up, a text model can take the routine spatial decisions.
The study covers simulated tabletops and regular shapes only. Sensor noise, heavy occlusion, and calibration drift are untested. Geometric estimation is a lightweight CPU routine. Learned 3D perception, articulated bodies, and deformable objects are not in the experiments. A one-shot shape initialization for a new category is left uncosted against later interactions.
A true support relation is a geometric candidate, not proof of physical stability. The synthesis gains are not split into a better program and an extra geometry API. Some token averages drop interrupted requests whose usage is unknown. Bridge-then-stack spends 6.7% less model time and 15.1% more tokens. Fewer images do not automatically make the episode cheaper. Tighter checkpoint formats are left for later.