Compressing Observation History into Agent Memory: Distilling Transformers into Recurrent Transformers
Philippe Weinzaepfel, Christian Wolf, Bülent Mert Sariyildiz, Guillaume Bono, Gianluca Monaci
cs.CV, cs.LG
2026-06-19
Chimera distills a full-history bottleneck into Kinaema. At 200 steps (2m/90°) it hits 70%, near the teacher's 72%, and 45% at 800 versus Kinaema's 37%.
An embodied agent that must answer "where is this image relative to me now" cannot drop old observations. A History Transformer attends over the full stream and is accurate, but storage and attention grow as O(T²). Recurrent Transformers such as Kinaema keep a fixed memory bank and update in O(1) per step. They also have to decide, right now, which parts of the current frame survive. Anything not written is gone.
Naver Labs Europe treats the accuracy gap as an optimization problem. Full-history models can wait for the query and then pick tokens. Recurrent models get no such hindsight, and the only supervision for a memory write travels through a long chain of later updates.
Chimera distills the teacher's compression policy into the student's memory.
The teacher is a Latent Bottleneck History Transformer (LBHT). Twenty learnable readout tokens sit on the full observation sequence and produce a 20×3072 bottleneck that matches Kinaema's memory tensor. The bottleneck is written before the query image arrives, so it behaves like a task-agnostic latent map, close to a Perceiver Resampler. Attention is non-causal: older tokens can be rewritten by later frames up to time t, never beyond. The teacher is not the deployed net. It has 7 layers, a masked-image auxiliary head, and random sequence lengths from 50 to 400, well past original Kinaema's 100-step training cap.
The student is Kinaema as published: 3 transformer layers, GRU gates shared across slots. Each step sees only the current 112×112 RGB (a finetuned DINO-v2 ViT-s) and a 7-d odometry delta. One student rollout is split into N segments. An L1 loss aligns student memory with the teacher bottleneck at each segment end, with truncated backprop. A relative-pose loss sits on the same endpoints, using both observed frames and simulator "alternative" views from already explored space. Student sequences go up to 200 steps. λdist is 0.001.
Two claims need to hold together: compressing a whole prefix is easier to learn than compressing it recurrently, and a recurrent update can track the teacher's bottleneck trajectory.
The task is memory-based relative pose estimation: given a query image, predict translation and rotation in the agent's current frame. Data come from Habitat on HM3D and Gibson. Test episodes shift the distribution, with sequences up to 800 steps, 25 cm forward motion against 10 cm in training, and 10° turns against 5°.
On RPE-test (HM3D val), 2 m / 90° accuracy:
| Method | 200 steps | 800 steps |
| LBHT teacher (full history) | 72 | 44 |
| Kinaema, same student net | 63 | 37 |
| Chimera | 70 | 45 |
| GRU | 56 | 31 |
| xLSTM | 47 | 29 |
Under the tight 1 m / 10° band, Chimera scores 36 against Kinaema 21 and teacher 44 at 200 steps, then 18 against 10 and 16 at 800. Past the teacher's 400-step training length, the recurrent student is slightly ahead.
Forcing Kinaema to T=400 for 600 epochs diverges. A bottleneck-free History Transformer is strong inside the training horizon (85 at 200 steps on RPE-val, 2 m / 90°) and falls to 35 at 800 steps. LBHT lands at 74 and 55, which is why it is the teacher. Raising distillation segments from 1 to 5 lifts the short-run 200-step 2 m / 90° score from 69 to 74, saturating near N=4. Causal teachers lag: 40 versus 76 at 200 steps under a matched 200-step / 300-epoch budget.
Memory statistics follow the teacher: smaller step-to-step updates, a stable core of slots, a few high-churn tokens. On an A100 the student spends about 7 ms per step with flat GPU memory. The teacher is quadratic.
This distills a compression policy, not a smaller network. Deployed parameter count and the compute graph equal Kinaema. For map-free pose, and for any setting that must recall an arbitrary past frame without a KV pile of images, that is the relevant comparison. Recurrent transformers may already be expressive enough; the missing piece is supervision for what to write.
The compute bill is large: about 30 days on one H200 for the teacher, 16 days on one H100 for the student. Evaluation stops at relative pose, with no navigation or manipulation loop.
Camera intrinsics and resolution are fixed. Training is simulation-only, because ground-truth poses and alternative query frames are hard to obtain from video.
The teacher bottleneck is non-causal while the student must run causally. The 800-step reversal may be recurrent inductive bias, or it may be teacher extrapolation failure; the paper does not separate those. All comparisons stay on Mem-RPE, with no head-to-head against CUT3R or VGGT-long. Distillation ablations use 200-epoch short runs, so their deltas should not be copied onto the main 400-epoch table.