Tool schemas are part of the model in agent evals
jonah_omninode · reddit · 2026-08-18
The author argues that tool definitions are often ignored in agent evals but actually change model reasoning. A reproducible eval unit must include tool names, schemas, defaults, and return formats. Semantic changes in tools require re-running evals to distinguish model regressions from harness changes.
More from coding & agent
- LLM-Powered CSV Parsing Boosts NEMT SaaS Adoption — threepointone · 2026-08-18
- AWS Launches Agents Pay Plugin for OpenClaw, Enabling Autonomous Agentic Payments — steipete · 2026-08-18
- Agent Governance Shifts to Device Level with mimOE Engine Release — shashib · 2026-08-18
- One prompt, ~2 hours: Claude + Thrixel autonomously build a full 3D game — RanaHanocka · 2026-08-18
- Permix: Lightweight Type-Safe TS Permissions Library Wins Developer Praise — jonathan_wilke · 2026-08-18
- Give agents a centralized source of truth before they hallucinate — eptwts · 2026-08-18