Efficient Test-Time Adaptation through Human-AI Interaction
Zora Zhiruo Wang, Apurva Gandhi, Rulin Shao, Aspen Chen, Jonas Mueller, Zhiqi Liang, Jett Chen, Michael Ryan, Qianou Ma, Luxi He, Zhoujun Cheng, Andre He, Seungone Kim, Jiayi Geng, Mingqian Zheng, Weiwei Sun, Zheyuan Zhang, Xinran Zhao, Yike Wang, Abe Hou, Liwei Jiang, Pang Wei Koh, Diyi Yang, Graham Neubig, Daniel Fried
cs.AI
2026-09-04
TAHI adapts agents from 20 sessions of human edits; solo success on 600 writing and viz tasks rises 4.5-20.9%, and evolved rubrics catch 16.0-22.3% more failures than LM-only ones.
Generalist agents already produce "good enough" drafts for writing, code, and figures. Professionals rarely sign those drafts as-is. Success criteria differ by person and are usually incomplete: users often discover what they wanted only after a few revision rounds. Those recurring edits, plan changes, and acceptance checks across sessions are mostly thrown away.
TAHI (test-time adaptation through human-agent interaction) treats that stream as training data. After each task, the agent updates its context or its weights, then faces the next task with a slightly better model of that one person. CMU, University of Washington, Stanford, Princeton, UCSD, and Handshake AI recruited 30 people and ran 600 tasks in two domains: paper-abstract writing and data visualization.
The backbone is Qwen3.6-35B-A3B. Context induction and rubric evolution use claude-sonnet-4-6. The UI, forked from open-source Agent Cowork, gives humans four channels: edit the plan, edit the deliverable (including drag-and-drop on figures, translated into code diffs), send a message, and edit the rubric. The agent acts through code: read and write files, execute, plan.
Adaptation is streaming. After task t the agent becomes A^{t+1} and then starts task t+1. Solo success is the first artifact of the current task, produced before any human input on that task, scored against the session's final evolved rubric.
Two update routes. Context adaptation has an LLM write declarative memory ("bold the title") and procedural skills ("abstract: motive, method, numbers") from human actions. Weight adaptation runs DPO on LoRA: the chosen trajectory is a collapsed one-shot version of the final draft, the rejected one is the first-round draft, with extra rejects sampled from the current policy. On-policy distillation and REINFORCE were tried; REINFORCE stuck near 0.3-0.4 success, and DPO won.
Each domain has 20 tasks built from 2025 NeurIPS, ICML, ICLR, CL, EMNLP, and CHI outstanding papers, plus 30 held-out tasks from 2023-2024. Writers are second-year-or-above CS PhD students; visualizers come from Handshake with data-analysis experience. Per domain: 5 people on context adaptation, 5 on weight adaptation, 5 on a non-adaptive offline control.
Within 20 sessions both routes raise solo success. Every paired t-test against the unadapted agent is significant.
| Route | Task | Baseline -> adapted (oracle) |
| Context | Abstract writing | 81.7 -> 85.4 (89.5) |
| Context | Data visualization | 77.0 -> 86.9 (95.0) |
| Weight | Abstract writing | 82.8 -> 86.5 (91.1) |
| Weight | Data visualization | 69.0 -> 81.1 (93.8) |
The paper reports those gains as +4.5% relative on writing, +12.9% for context visualization, and +20.9% for weight visualization. On held-out tasks, context reaches 93.5 / 92.1 and weight 93.0 / 91.2. Visualization gains shrink off the training set; writing holds up, consistent with a backbone that already writes reasonably well.
Context induction from messages alone reaches 81.9 writing and 68.0 visualization. Adding file edits, plan edits, and rubric edits lifts that to 84.9 and 75.6: 3.0 and 7.6 extra points from non-text actions. Weight adaptation is cheaper at inference: 62.6% and 88.3% fewer input tokens than context adaptation, and slightly shorter outputs. Context adaptation inflates message length by 143%-176%.
Surface form and naming are easy: 70.4%-73.7% of those writing rubrics get absorbed. Problem framing and context specificity are not, at 41.9%-43.4%. On visualization, HTML conventions hit 94.4% and data fidelity 75.2%, while "what to highlight, and how" still leaves a 50.0%-60.1% gap to the human oracle.
Evolved rubrics score imperfect solo drafts at 82.3% (writing) and 66.4% (visualization). LLM-only rubrics sit at 98.3% and 88.7%, missing 16.0-22.3 points of failure. About 67.9%-70.7% of rubric items are shared community norms; the rest are personal. After per-user adaptation, shared items still rise by up to 8.8%, and personal items rise more. Merging several personalized agents still helps visualization by about 3 points and hurts shared writing items.
Personalization does not have to wait for tens of thousands of preference pairs. A few dozen real revision sessions already move a test-time agent toward one person. Product teams building copilots should treat plan edits, file edits, and rubric edits as first-class training signal, not throw them away after the chat. Weight adaptation saves context but needs a LoRA training path. Context adaptation is inspectable and editable, and it makes the agent wordier.
The evolving rubric may outlive the adaptation algorithm. Open-ended work still lacks a decent acceptance test; this is one way to grow one from real edits with an LLM drafter and a human in the loop. Merging many personal agents into one community agent is unsolved. Do not expect a single global model to swallow every taste.
There is no dedicated limitations section. The holes are visible. The tasks are abstract writing and HTML figures, both artifact-centric, and they do not stand in for coding, research, or browser use. Each cell has 5 people. Test-time success is scored on the same-session final rubric, which has already seen human interaction on that item. Held-out numbers are the cleaner generalization evidence, and those rubrics are themselves LLM-written from a user's general rules plus a 20-task summary.
Oracle scores often miss 100%; the authors blame model skill and how much effort humans were willing to spend. Open-source models failed, in pilots, to write reusable memory, skills, and criteria, so induction is tied to Claude. The conclusion says "20 experts"; the abstract says 30 people. The gap is 5 offline controls per domain. The claimed 20.9% relative lift on weight visualization does not match Table 1 arithmetic on 69.0 -> 81.1 (about 17.5%). Cite the absolute scores.