Tweaking Tool Output Format Lifted Agent Rename pass@1 from 0.67 to 0.83
RunAI_Coder · reddit · 2026-09-07
A small pilot found coding agents almost never pick LSP tools voluntarily—models chose findreferences 0-6% of the time on code-location tasks, defaulting to grep. The striking part: keeping the same backend and reference set but attaching ±2 lines of source to each LSP result boosted multi-file rename pass@1 from 0.67 to 0.83 and cut follow-up file reads from 15.2 to 3.2 per episode (below grep's 4.3). LSP still wins on noisy repos (+0.246 F1, 12% fewer tokens) but adds nothing on clean ones, and grep covers docstrings/config strings LSP excludes. A companion Spotify case: CLAUDE.md rules to route big reads to a smaller model proved "advisory, not enforced," so they shipped a PreToolUse hook blocking Read on files over 350 lines. Lessons: return context, not bare addresses; enforce tool routing with hooks rather than instructions.
More from coding & agent
- Google AI Search MCP server adds grounded web search via Vertex AI or Gemini — modelcontextprotocol · 2026-09-07
- Asterwise launches Vedic astrology MCP server with real ephemeris calculations — modelcontextprotocol · 2026-09-07
- CubeSandbox v0.7.0 ships cross-node pause & resume for AI agent sandboxes — HeyAmit_ · 2026-09-07
- GPT-6 Astra turns one prompt into a 1440p Colosseum 3D video via Blender MCP — Scobleizer · 2026-09-07
- awesome-ai-apps: open-source repo with 132 LLM app examples, agents and RAG demos — Arindam_1729 · 2026-09-07
- Why 88-95% of enterprise AI agent pilots never ship — and what working teams do differently — ankitsharma112 · 2026-09-07