DRFLOW: EMNLP benchmark shows deep research agents struggle at personalized workflow prediction
mageed · x · 2026-10-27
ServiceNow AI Research and UBC present DRFLOW at EMNLP2026, a deep research benchmark for personalized workflow prediction — enterprise tasks requiring an ordered list of steps for a specific person rather than a report.
The benchmark has 100 tasks across B2B, B2C, education, healthcare, and legal domains, with 1,246 reference steps grounded in 3,900+ sources and 7 diagnostic metrics; a mixed split adds distractor artifacts.
Key results: the DRFA workflow agent scores 57.9 F1 vs 47.9 for the DRBench baseline (averaged over four LLM backbones); best is 63.9 with GPT-5.2 — far from solved. Condition resolution is the bottleneck (38.2, only +20.2 over baseline), and mixed tasks drop all agents to 43.9. Both agents are highly factual (87.8 vs 89.8) — the gap is in the steps.
More from Research
- Physicist on OpenAI's math result: it's mathematical physics, not theoretical physics — skdh · 2026-10-27
- Tsinghua team open-sources C2C: LLMs swap KV-Caches directly for 2.5x faster, more accurate collaboration — solyarisoftware · 2026-10-27
- Qwen model has a dedicated MoE expert for Chinese cloud concepts, sparking URL hallucinations — builderjaydub · 2026-10-27
- OpenAI preprint claims 3-SAT solvable in sub-exponential time, challenging ETH — cephaloform · 2026-10-27
- Pathology foundation models hit AUROC 0.90 vs ResNet50's 0.81 — maier_ak · 2026-10-27
- TUM study: 19 pathology foundation models find cancer but miss dysplasia 48% of the time — maier_ak · 2026-10-27