HALO: a human-grounded benchmark judging proactive agents on timing, not just correctness
muhao_chen · x · 2026-11-06
The 2077AI team released HALO, a human-grounded benchmark for proactive agents built around a key insight: acting correctly isn't enough if the intervention is poorly timed or unwarranted.
Structure
- HALO-Arena: everyday apps (email, calendar, chat, contacts, wallet) across 10 domains and 4 need types (dependency/blockage/risk/opportunity); agents retrieve evidence from profiles, text files, and databases before deciding to act
- Intervention levels: L0 observe → L1 notify → L2 prepare → L3 execute, testing how far an agent should go
- HALO-8K: 50 environments, 400 scenarios, 8K agent answers, verdicts labeled by 20 experts
Evaluation dimensions: beyond appropriate/inappropriate verdicts, failures are diagnosed by timing (premature/belated), intensity (insufficient/excessive), and consumption (reasoning tokens, retrieval rounds).
More from coding & agent
- Google donates Agent Substrate, gVisor and OpenRL to CNCF; Agent Substrate hits 1.0 on GKE — HowDevelop · 2026-11-06
- Deep-dive report on parallel coding agents: worktree isolation doubles costs, merge conflicts dominate — weswinder · 2026-11-06
- OpenAI's Justin Uberti to reveal GPT-Live speech-to-speech architecture at VapiCon 2026 — juberti · 2026-11-06
- Approved action ran twice in 3 agent frameworks: crash-resume test — KaleidoscopeNew9259 · 2026-11-06
- Before automating Shopify flows, run a dry run listing every product and price it would change — shashib · 2026-11-06
- Melite open-sources agentic ComfyUI: agents that generate and extend their own harness — data-cypher-000 · 2026-11-06