File-Retrieved Tool Output Makes GPT Agents Substitute Approximations, Costing Accuracy
ucriverside · hf · 2026-10-23
Researchers at UC Riverside and UC Irvine studied a common agent harness design: writing long tool outputs to files for on-demand retrieval instead of stuffing them into context. They used USGS's geochemistry simulator PHREEQC — reports up to 846.8k characters and fully deterministic — and had agents answer 158 execution-verified multiple-choice questions.
Key findings
- Even when the underlying computation is correct, agents can answer wrongly: when retrieving long simulator outputs from files, GPT models substitute their own approximations for authoritative printed values
- The harness design itself induces accuracy loss despite correct computations
- Prior work (Sen et al., 2026) found file-based delivery lowers accuracy across most harness–model pairs without explaining why; this paper pinpoints where the loss arises, since a wrong answer rarely reveals whether the computation or the reading was at fault
More from coding & agent
- Personal AI agents become a capital-intensive race as cloud execution environments reshape cost structures — sujingshen · 2026-10-23
- LLMs make code so cheap that recursive rewrites could make verified software tractable — jd_pressman · 2026-10-23
- Hugging Face hands OpenEnv RL environments project to Linux Foundation for neutral governance — lmoroney · 2026-10-23
- Theo writes the definitive explainer on what an AI coding "harness" actually is — pvncher · 2026-10-23
- Conga launches MCP Server letting Claude and Copilot query live contract data — shashib · 2026-10-23
- Dev who shilled GPT for a year says Claude Opus 5.5 is 'so nice to work with' — rudrank · 2026-10-23