onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction
Lei Yang, Mengyin Liu, Jia Wang, Hangyu Guo, Liang Zhao, Zheng Ge, Kang An, Binxing Jiao, Qi Han, Daxin Jiang, Siqi Shen, Xiangyu Zhang
cs.CL, cs.HC, cs.LG
2026-09-22
onPanda lets annotators fix the first bad token then continue. On 21 image-caption prompts, median time drops 52% vs post-editing; PPL is only +0.86% off the rollout.
Alignment data and agent trajectories keep failing the same three-way tradeoff: cost, on-policy fidelity, and supervision granularity. Writing or post-editing a response can meet an SFT bar, but it is slow, and the finished text leaves the rollout model's sampling distribution. Preference ranking is cheaper, yet it only labels whole responses. Candidates must already be samplable. If the model cannot produce a good answer, ranking cannot inject one.
Agents make this worse. A trajectory is a multi-step loop with tools. A correction has to actually run the tool, get real feedback, then continue. Existing platforms can watch, score, and interrupt live agents. Reptile, aimed at terminal software-engineering work, lets annotators edit model text and execute it, but almost every fix is typed by hand and the current design cannot handle general tool-call formats. What is still missing is an interactive tool that, across environments, can correct reasoning and tool calls in real time and then execute the corrected calls.
onPanda's core loop is token-level correction. The annotator reads the rollout, finds the first bad token, and either clicks a substitute from the model's top-20 candidates at that position or double-clicks and types free-form text. The system truncates everything after that point and continues generation from the corrected assistant prefix. The locate-correct-continue cycle repeats until the response passes the SFT quality bar.
Errors often start at one fork and then pollute the rest of the reply. Fix the first bad token and let the model continue, and later copies of the same hallucination often disappear. Humans touch a few positions; most tokens still come from the rollout model. Picking a candidate perturbs the distribution less than typing. Free-form edit is the fallback for directions the model cannot sample.
The inference API must support two things: continuation from an assistant-message prefix (the paper cites vLLM's continuefinalmessage) and per-token top-k logprobs. The UI groups tokens into grapheme-level chunks so people can click readable units; records stay token-precise. A session is an annotation tree: each correction forks a new node, and intermediate versions are kept automatically. Nodes marked isgood=Y export as SFT. Positive and negative nodes under the same prompt become preference pairs. Each correction also yields a triple (negative sample, rejected token and position, chosen token) that can feed DPO or a process reward model.
For agents, a response template maps both ways between structured messages (reasoning, content, toolcalls) and the model's native token stream, so special tokens such as </think> and <|toolcallbegin|> are visible and editable. Tools connect through MCP. Claude Code, Codex, and OpenClaw wrap into MCP servers. Tool calls can wait for approval: the annotator may fix arguments then run them, or reject with optional text; rejected trajectories become negatives. Tool results go back into context and generation continues. Image, audio, and video messages use the same interface.
The system is a front-end component library. It can run as a static page with no database, talking to a Chat Completions API from the browser, or sit inside an existing data platform. Artifacts land in a single .panda.json file.
The controlled study is small: three annotators, 21 image-description prompts, Latin square so no one labels the same prompt twice. All three workflows share the same first rollout from Qwen3.5-35B-A3B in instruct mode (temperature 0.7, top-p 0.8). Baselines are post-editing in POTATO and four-way ranking in Argilla. None of the annotators are onPanda developers, and all three methods got warm-up tasks.
| Tool | Paradigm | Median / mean time | Pairwise win | PPL (Δ vs 1.171) | SFT coverage | Pref. pairs |
| Argilla | rank 4 | 336s / 685s | 28.6% | 1.161 (−0.83%) | 52% | 6.00 |
| POTATO | post-edit | 681s / 711s | 54.8% | 1.596 (+36.31%) | 100% | 0.95 |
| onPanda | token-level correction | 330s / 516s | 66.7% | 1.181 (+0.86%) | 100% | 7.43 |
Median time is 51.5% below POTATO and essentially tied with Argilla. Mean time is lower than both, because Argilla's hard prompts require reading four long candidates and the mean blows up. Per-prompt PPL already swings about ±2.8% across rollouts; onPanda and Argilla sit inside that noise, while post-editing lifts PPL by 36%. The LLM judge is GPT-5.5 with order swapped. Humans preferred onPanda over POTATO in 54.8% of pairs. Adapted NASA-TLX workload (0–10, lower is lighter): 3.1 vs Argilla 5.4 vs POTATO 6.8.
Production logs 25,596 vision, 105,143 audio, and 1,257 agentic sessions, about 388K token-level corrections. Of tokens in accepted responses, 97.0% were model-generated, 2.1% came from candidate clicks, and 0.9% were typed. Median session time is 31.3 minutes for agents versus 1.65 and 1.93 minutes for vision and audio; agent trajectories also need more typed edits, which the authors blame on longer traces and still-unstable rollouts.
Panda-CVL is a mostly Chinese vision-language release: 7,491 sessions (6,839 train / 652 test), annotated against step-1o-turbo, a 32B dense VLM. The benchmark expands the test set into 2,126 instances (652 good, 1,474 not-good) and asks a model to accept, or locate and replace the first error. Best F1 is 17.09% (GPT-5.5). GPT-6 leads localization at 24.46% and end-to-end correction at 15.83%. Humans do not agree that tightly either: four independent annotations of the same initial response match exact first-error position 30.95% of the time, 44.44% with a four-token window. When the position matches, replacement-token agreement is 69.44%.
For teams that collect alignment data, this is a human-in-the-loop continuation editor. You keep 100% SFT coverage without dragging the text 36% off-policy the way full post-editing does. Preference pairs grow out of the correction tree; you do not need a separate ranking pass. Token-level triples are position-precise and naturally paired, which could be a cheaper DPO or PRM signal. This paper does not train on them, so that remains a claim.
The more concrete increment is agent annotation: edit tool-call arguments, approve execution, feed real tool results back, continue. That is closer to a live rollout than rewriting a finished trajectory. The API tax is real. Open-weight stacks (vLLM, SGLang) already expose prefix continuation and logprobs. The authors list Doubao, DeepSeek, Kimi, and StepFun official APIs as supporting both. APIs that hide those two features cannot produce on-policy data in this tool.
The Panda-CVL numbers are low enough that "find the first bad token and fix it" is still a hard task. Treat the release as a tool and a data protocol, not as a finished training method.
The authors list hard constraints. The interaction depends on prefix continuation and top-k logprobs; some proprietary APIs expose neither. Efficiency and on-policy gains assume sparse errors. If the model is far from the task and every sentence needs a fix, both advantages collapse and annotators fall back to typing. On-policy is relative to the rollout weights at annotation time. Train a different model, or keep updating the same one, and that property fades.
The controlled experiment is 3×21, one image-caption task, one rollout model, all in-house annotators. Quality mainly uses GPT-5.5 as judge; the human check only covers onPanda versus POTATO. There is no controlled agent study, only system capability and production stats. The largest gap is downstream: they measured annotation time, output quality, and distributional closeness, and they say outright that they have not run training experiments on the token-level signal.
Human location agreement is 31% exact. Single-reference Corr.-NG will be capped by annotator disagreement. Read the 17% F1 as a noisy ceiling, not as a pure model failure. The comparison with POTATO and Argilla also evaluates whole workflows, so the paper cannot separate the interface from the paradigm.