PlanBench-XL Teaches Agents to Discover Tools
Shahules786 · x · 2026-07-14
The author introduces the approach of PlanBench-XL: processing tool information via a retrieval interface instead of stuffing all tool definitions directly into the context.
Design Highlights
- Tool definitions are stored in files
- Agents must repeatedly retrieve required tools during tasks
- This closely mimics enterprise environments with massive toolsets that cannot fit entirely into the context
The author also mentions they are experimenting with "fully visible / partially visible" modes for the same tasks in their own environment, aiming to simultaneously benchmark and train agents to actively explore the tool library.
Related event: PlanBench-XL Tackles Agent Tool Retrieval at Scale(3 posts)→
More from coding & agent
- BELL: Open-Source Stack Combines Browserbase, Eve, iMessage and Stripe Link for Personal Agents — jeff_weinstein · 2026-10-27
- Two engineers rebuilt a FedEx supplier's billing core in 3.5 months vs an estimated 7-8 — alex_verem · 2026-10-27
- GPT-6.1 Terra gains ~20 points on Terminal-Bench 4.0, rivalling GPT-6.1 Sol — ArtificialAnlys · 2026-10-27
- GPT-6.1 Terra scores 48 on AA Coding Agent Index at one-third the cost of GPT-5.6 Terra — ArtificialAnlys · 2026-10-27
- Shift-1 router cuts coding agent spend up to 55% while keeping SWE-bench accuracy — ypatil125 · 2026-10-27
- Non-coder vibe-codes a Godot JRPG with Claude Code in two weeks, music and art synced — ADHDavid · 2026-10-27