Flexible Language-Guided Representation Editing for Adaptive Robot Behavior
Audrey Lee, Nathaniel Dennler, Ilia Sucholutsky, Andreea Bobu
CoRL 2026)
cs.RO
2026-10-21
FLEX adds, generalizes, and masks robot features. Across seven navigation mismatches, normalized reward is at least 0.91, and 12 users rate intent above add-only ALGAE (p=0.006).
A coffee-carrying demonstration can arc around a laptop. It cannot say whether the thing to avoid is that laptop, every electronic device, or whether the wooden bowl on the table counts. A trajectory records how the arm moved. The features a person actually cares about are measurable properties of the scene, such as distance to electronics. Fit the weights on the wrong features and the motion is still wrong.
Reward is a linear mix, R(τ) = θ⊤Φ(τ). The weights θ can be recovered from demonstrations with maximum-entropy inverse reinforcement learning. The feature set Φ is usually fixed before deployment. Prior editors mostly add features and stop. After several rounds, a feature that should have become a category stays pinned to one object, and distractors remain. A representation fails in three ways: a missing feature, the wrong level of abstraction, or an irrelevant quantity. Add-only editing leaves the last two alone.
FLEX (Flexible Feature Editing Pipeline) changes Φ from natural-language corrections and changes θ from demonstrations. Each round the robot rolls out a trajectory under the current Φ and θ, and the person describes the gap in a sentence. The language model is GPT-4o throughout. It rewrites the feature set and does not touch the weights. By assumption, a correction says which features matter and says nothing about how much. A line like stay even further from the laptop keeps the laptop and drops the word further.
The edits run in a fixed order, add, then generalize, then mask. Each step reads the set the previous step just wrote.
θ is then re-estimated from demonstrations on the new Φ, and the robot replans. One sentence can ask for a new feature, a wider one, and a deletion together, which is why the edit is split into three steps.
The metric is normalized reward: a trajectory return scaled to 0 for the worst-case path and 1 for the optimal path under the true reward. Start, goal, and object poses are redrawn over five seeds.
2D navigation is a 9×9 grid. The kitchen avoids a stove and a kettle, with the fridge as a distractor. The lava map avoids swamp and lava, with meadow as the distractor. Seven columns cover one mismatch type, each pair, and all three at once. ALGAE only adds features. MFE starts from an exhaustive list and masks. USF skips the split and emits a whole new Φ from the correction, the demonstration, and the current set.
| Method | Add only | Generalize only | Mask only | Add+Gen | Add+Mask | Mask+Gen | All three |
| ALGAE | 1.00±0.00 | 0.59±0.02 | 0.95±0.03 | 0.77±0.02 | 0.59±0.02 | 0.63±0.02 | 0.69±0.03 |
| MFE | 0.95±0.01 | 0.70±0.02 | 1.00±0.00 | 0.75±0.02 | 0.43±0.01 | 0.67±0.02 | 0.83±0.02 |
| USF | 0.64±0.06 | 0.72±0.03 | 0.90±0.02 | 0.80±0.01 | 0.99±0.01 | 0.78±0.04 | 0.78±0.03 |
| FLEX | 1.00±0.00 | 0.91±0.03 | 1.00±0.00 | 1.00±0.00 | 1.00±0.00 | 1.00±0.01 | 0.99±0.01 |
ALGAE scores 1.00 when the only needed edit is an addition, 0.59 when the needed edit is generalization, and 0.69 when all three mismatches are present. MFE scores 1.00 on mask-only and 0.43 on add-plus-mask. USF scores 0.64 on the simplest add-only case and 0.99 on add-plus-mask. FLEX's lowest cell is generalize-only at 0.91±0.03. The other six sit between 0.99 and 1.00.
On a 7-DoF Franka arm, corrections stack across five stages: carry coffee clear of electronics (a wooden bowl is the distractor), carry a lit candle clear of flammables, or carry a bowling ball clear of fragile objects. Objects change between stages and Φ carries forward. The paper plots normalized reward and does not list point values. FLEX matches the true reward after one or two corrections and stays there. The baselines climb more slowly, then fall as spurious features pile up and earlier features are never re-abstracted.
Twelve university participants (6 women, 6 men; five reported almost no robot experience) used FLEX, ALGAE, and MFE within subjects, one Franka scenario each, order counterbalanced. They typed five corrections per method, steering the robot toward a preference fixed in advance, with that preference's goal trajectory on screen. Interaction stayed in simulation because some baselines moved the arm erratically. The sentences were later paired with expert demonstrations from the physical Franka and the reward was fit offline. The figure matches the simulated sequences: FLEX holds, the baselines decline, and point values are not printed. On 1-7 Likert items, Holm-corrected pairwise Wilcoxon tests put FLEX above ALGAE on task performance (p=0.035), intent understanding (p=0.006), and robustness to unintended consequences (p=0.047), and above MFE on intent understanding (p=0.031). Robustness versus MFE is p=0.074. Ease of use and collaborativity are highest for FLEX, and those gaps are not significant. Means are not tabulated.
Online reward learning needs a short feature list, because deployment does not yield many demonstrations. FLEX maintains that list from a few sentences, rather than an expert catalog or an add-only set that only grows. Feature code comes from the language model. Weights still come from maximum-entropy inverse reinforcement learning. The planner can stay put.
The gain is incremental. ALGAE already reaches 1.00 when a single missing feature is the whole problem. Across five object changes, the set also has to widen laptop into electronics and drop the bowl, or Φ bloats and overfits the scene in front of it. The operating region is narrow: objects need text attributes, the preference has to look like shared commonsense, and the motion has to be quasi-static manipulation of rigid bodies.
Four boundaries are stated up front. Dynamic tasks and deformable objects may not be specifiable in language alone. Commonsense covers shared preferences and fails on counterintuitive or purely personal ones. When one sentence encodes both Φ and θ, the weight part is discarded by assumption. The closed-loop study ran in simulation. Hardware is an offline check, not a person correcting a physical arm in the loop.
The study is narrower than those four lines. Participants matched a scripted preference while looking at the goal trajectory, so the test is whether they can say a difference they can already see. The physical-robot curve pairs expert demonstrations with participant language, two noise sources that never showed up in the same interaction. MFE starts from an exhaustive feature list, so its 1.00 on mask-only is not a same-start comparison. Only GPT-4o was tested, and there is no separate success rate for whether the generated feature code ran. The Franka and human results are figures, so those gaps cannot be cited as a precise delta.