Under matched token budgets, most gains of web agent online augmentation vanish
hllo_wrld · x · 2026-10-23
A new study (arXiv:2606.15017) re-evaluates recent online augmentation methods for web agents—AWM, ASI, and ReasoningBank—on WebArena against a vanilla agent given the same total inference budget, which it may spend on more interaction steps. Under matched budgets, much of the reported advantage of all three methods disappears.
Two takeaways: compare agents at equal token count rather than equal step count, and scaffolding for its own sake often yields poor performance per token spent. Practitioners in the replies report observing the same effect.
Related event: Web Agent Online Enhancements Lose Most Gains Under Equal Token Budgets(2 posts)→
More from coding & agent
- 7 Mistakes Economists Make With Claude Skills: Not Building Their Own and Making Too Few — aniketapanjwani · 2026-10-23
- MIT Sloan research: AI speeds up coding but delivery still bottlenecks on testing — sven_ai · 2026-10-23
- Agents logging into bank accounts violate most banks' terms, even with user consent — pstAsiatech · 2026-10-23
- Codex desktop shipped with no editor: 4-person team bet everything on agents — sujingshen · 2026-10-23
- First company officially incorporated by an AI agent via Stripe's Atlas MCP — jeff_weinstein · 2026-10-23
- Designing a full motion system in one weekend with AI, in low-effort mode — soleio · 2026-10-23