Paper: Vulnerability-Hunting Agents Burn 60.4% of Tokens Just Understanding Code
rohanpaul_ai · x · 2026-10-19
The arXiv paper "Where Do the Tokens Go? Understanding and Reducing Costs in LLM Agents for Vulnerability Discovery" diagnoses where security agents' token budgets go, via an open-coding study of 200 CyberGym traces across four agents (Codex, OpenCode, Cybench, EnIGMA).
Key findings:
- Agents vary substantially in success and cost; higher spending doesn't consistently yield better outcomes.
- Code localization/understanding plus vulnerability reasoning and trigger design consume 60.4% of tokens and are the two leading bottlenecks in failed runs — agents can burn millions of tokens without producing a working PoC.
- Only 24.4% of matched comparisons preserve success at lower cost with existing efficiency methods; unsuitable signals and auxiliary overhead limit their benefits.
The authors propose AVRI, an Agent-centric Vulnerability Reasoning Interface built on a persistent Bidirectional Evidence Trace (BET) that links how the harness consumes input to the conditions making an operation unsafe, helping agents build and maintain reasoning context to cut wasted tokens.
More from coding & agent
- State machines for agent logic are extremely slept-on: LEGO bricks agents can rearrange — DavidKPiano · 2026-10-19
- Data preparation is architecture: a six-step pipeline shared by fine-tuning and RAG — Puzzleheaded_Box2842 · 2026-10-19
- Inspired by Booking.com's MCP playbook, author launches a Query Agent MCP server — CShorten30 · 2026-10-19
- Booking.com's platform team distributes MCP servers so internal teams can build tool-calling agents — CShorten30 · 2026-10-19
- Hunting for an agent that auto-tops up LLM API balances across providers — huangyun_122 · 2026-10-19
- LangChain CTO to join fireside chat on turning production agent traces into better models — hwchase17 · 2026-10-19