DocOps benchmark finds frontier agents still fail on long-horizon document tasks

Jiazhen Jiang · hf · 2026-07-23

DocOps introduces a verifiable benchmark for autonomous agents that operate on complex documents.

• The paper defines a hierarchical taxonomy that breaks document work into atomic actions and increasingly complex workflows.
• It evaluates representative closed- and open-source models across multiple agent harnesses.
• The authors find that even frontier systems still struggle with long-horizon, tightly coupled document tasks.
• Three recurring failure modes stand out: long-term state tracking collapse, shallow semantic verification, and destructive edits to structural metadata.
• The work is framed as a step toward more robust, non-destructive agents for real document-heavy workflows.

Original post →

More from Research

• Diamandis: AI Cancer Navigator and $100 Genome Sequencing Close the Care Gap — PeterDiamandis · 2026-10-27
• EMNLP Oral: training-free Nexus Sampling cuts KV-cache memory 10x; OpenResearcher tops BrowseComp-Plus at 54.8% — yuz9yuz · 2026-10-27
• ICML 2026 position paper: LLM 'thinking traces' are a harmful anthropomorphism — rao2z · 2026-10-27
• Forking gap follows a power law in context frequency; masking rare n-grams shrinks it — ZimingLiu11 · 2026-10-27
• How forking happens: n-gram tables split logits into non-interacting subspaces — ZimingLiu11 · 2026-10-27
• The best LoRA explainer video yet: rank constraints meet intrinsic dimension — sophiamyang · 2026-10-27

Research channel →