RuleTrek benchmark: best of 11 frontier LLMs scores only ~62% on agentic rule reasoning
Chenhao Yuan · hf · 2026-10-15
Researchers from the Chinese Academy of Sciences and BAAI introduce RuleTrek, a multi-domain benchmark for agentic rule reasoning in complex, multi-file scenarios.
- Existing rule-reasoning benchmarks represent scenarios with only a few factual statements, oversimplifying rule identification and application
- RuleTrek extracts structured rule graphs and incrementally composes rule-specific sub-scenarios while ensuring consistency and solvability
- It evaluates three capabilities in interactive workspaces: Rule Identification (RI), Rule Deduction (RD), and End-to-End reasoning (E2E)
- Across 11 frontier LLMs, the best RI and RD scores are only 62.47% and 60.43%; E2E scores range from 26.97% to 55.29%
The takeaway: current LLM agents struggle substantially with rule reasoning over realistic, distributed scenarios. Code and dataset are public on GitHub.
More from coding & agent
- shwarm.org: like a swarm of agents, but with humans in the loop — allisonmaybe · 2026-10-15
- How production Text-to-SQL systems catch semantically wrong queries that run just fine — Fun-Following-1723 · 2026-10-15
- Tencent ARC's Player-1 learns general game control from synthetic experience — _akhaliq · 2026-10-15
- Linus Torvalds slams AI bans as 'UTTERLY STUPID': Linux kernel won't restrict AI tools — ducha_aiki · 2026-10-15
- Memory API for AI agents: Wontopos claims 95.7% on LongMemEval-S, seeks pricing feedback — True_Mongoose_7073 · 2026-10-15
- Capture intent before model invocation: a human-AI loop where AI only enters at refine — lee_stott · 2026-10-15