Terminal Agents: A Survey of AI Agents in Command-Line Environments
Yi Bin, Xiaoyang Yuan, Haoxi Zeng, Wencheng Ye, Wenqi Shao, Chen Qian, Wei Ye, Yujuan Ding, Zheng Wang, Pengpeng Zeng, Jingkuan Song, Heng Tao Shen
cs.AI, cs.SE
2026-08-21
Tongji survey defines terminal agents by command-mediated loops and seven competences. SWE-agent beats mini by up to 21.25 points; Flash vs Pro by at most 2.50.
LLM agents increasingly edit files, install dependencies, run tests, and read logs through a terminal. Existing surveys split that behavior across software-engineering agents, tool use, and GUI or browser computer-use. They never treat "does task progress actually come from command execution" as the cut.
Repository repair on SWE-bench, command workflows on Terminal-Bench, and deployed tools such as Claude Code or Codex CLI then look like unrelated stories. This survey takes the terminal as the execution substrate, draws a workload-level boundary, and uses a seven-dimension competence profile to talk about architecture, learning, and evaluation in one vocabulary.
A workload counts as a terminal agent under three tests:
Occasional shell calls and one-shot patch generation without iterative execution are out. SWE-agent and terminal-centric OpenHands workloads are in. Static patch pipelines, GUI agents, and CLI-packaged assistants that only use the terminal as an access surface are out.
The seven dimensions split by what is being controlled, what evidence the next decision needs, and what happens when execution diverges: command and action formulation; feedback and artifact interpretation; runtime and environment management; state, task, and context tracking; progress verification; recovery and adaptation; governance and side-effect control.
Architecture is four layers: interface and observation; runtime and workspace; control, verification, recovery, and governance; harness and context. Realized behavior is jointly shaped by the model, the interface, the harness, the runtime, and the environment. Reporting a model name is not a system description.
The research corpus covers 2022 through August 2026: 186 research entries, 112 of them from 2026, plus deployment comparators such as Claude Code, Codex CLI, Aider, and Gemini CLI.
Several literature findings already converge. Executable trajectories are what turn verification and recovery into learning signals, yet current training still concentrates on successful repository traces. Leaderboards still anchor on final pass rate and expose process quality, recovery, and governance unevenly. SWE-EVO reports a drop from 65–73% on SWE-Bench Verified to 21–25% on multi-file evolution. An audit of five terminal-agent benchmarks finds 16% of tasks reward-hackable. WildClawBench reports an 18-point harness-conditioned gap.
The authors add two bounded fixed-condition diagnostics. The first freezes mini-SWE-agent with DeepSeek-V4-Flash and varies four benchmark families:
| Benchmark | Tasks | Official score | Final-window verify P5 | Governance trigger P7 |
| Terminal-Bench 2.1 | 241 | 52.6% | 29.5% | 0.5% |
| SetupBench | 93 | 59.1% | 36.6% | 1.4% |
| LongCLI-Bench | 21 | 23.8% | 23.8% | 4.5% |
| BashArena | 640 | 41.8% | 66.2% | 3.0% |
Rule-matched invocation failures (P1) round to 0.0% on all four, so that signal is too narrow to catch semantic action errors. Final-window verification and governance triggers differ by multiples across families.
The second diagnostic compares three complete systems on identical task IDs. On Claw-SWE-Bench Lite (80 tasks), SWE-agent resolves 76.25% (Flash) / 77.50% (Pro), mini-SWE-agent 58.75% / 56.25%, OpenHands 68.75% / 67.50%. The largest system gap is 21.25 points. On SWE-bench Lite (300 tasks) the same three systems differ by at most 7.00 points, and the OpenHands vs mini ranking reverses. Inside a fixed system, Pro minus Flash never exceeds 2.50 points; paired tests are not significant.
For people building coding agents, the paper writes down a fact already visible in production: a score is a coupled product of model, interface, harness, runtime, and budget. In their snapshot, swapping the system moves 21 points; swapping Flash for Pro does not move 3. Papers that name a model and omit the harness and runtime conditions are not comparable.
For evaluation design, repository repair is not terminal competence. Setup, long-horizon, and safety suites light up different dimensions. No single leaderboard substitutes for process evidence. Replayable traces, fresh tasks, and protocols that score governance together with task success are the actionable agenda.
Each diagnostic cell is a single formal run; no multi-seed variance is reported. Only two DeepSeek-V4 variants appear, so the Flash/Pro result does not travel to other model families. P1 being uniformly zero shows that the process metrics are still coarsely operationalized. LongCLI-Bench has 21 tasks, so its auxiliary feedback, state, and recovery rates should be read directionally. The corpus and the empirical base remain software-engineering heavy; evidence from operations, data, scientific workflows, and security is, by the authors' own calibration, fragmented. The survey proposes no new algorithm. The companion GitHub list is a literature index, not a reproducible experiment suite.