RealCLI benchmark unifies CLI agent evaluation with human-agent collaboration quality
IQuestLab · hf · 2026-10-29
RealCLI, a benchmark from Beihang University, Renmin University and IQuest Research (arXiv, Oct 15 2026), is the first paired benchmark unifying CLI agent task execution and human-agent collaboration quality:
- Framework: five dimensions operationalized into 22 sub-dimensions and 51 interaction scenario templates on realistic CLI tasks
- Paired modes: REALCLI-EXEC supplies complete specs upfront; REALCLI-COLLAB adds a scripted user delivering progressive information, corrections, constraint revisions, and authorization decisions mid-run
- Findings: major gaps in steerability, trust calibration, and safety management across frontier models
Existing benchmarks assume fixed instructions or judge only outcomes; RealCLI captures runtime collaboration. Code and data are on GitHub and Hugging Face.
More from coding & agent
- Dev's code merged by AI agent with zero human intervention — iamsahaj_xyz · 2026-10-29
- Turning a laggy old Xiaomi phone into an offline MP3 player with Claude — PeanutGrinderZ_2091 · 2026-10-29
- 10 open-source repos that give coding agents a code map, led by 125k-star Graphify — victor_explore · 2026-10-29
- Snaptrude launches AI agent that takes building designs from site to construction-ready LOD 400 models — EXM7777 · 2026-10-29
- DecisionRAG replaces the retriever with a decision model, beating 8B rerankers by 13.8 R@1 — omarsar0 · 2026-10-29
- Turning photos, charts and logos into animated videos in 5 minutes with Claude Code and Remotion — TawohAwa · 2026-10-29