SilentGuard: harness-level invariant checks cut agent API silent-failure errors from 74% to 1.7%
Genentech · hf · 2026-11-18
Genentech's SilentGuard tackles silent API failures in LLM agents (e.g. a misnamed filter returning HTTP 200 with a plausible but wrong result). Instead of asking the model to catch errors, it enforces invariants at the harness level, testing each API result and withholding values that violate them.
Key findings:
- 11 of 15 public research APIs show silent failures, with counts inflated up to 691x
- Existing fixes (checklists, reviewer models, LLM judges) leave over 70% of answers wrong
- Across 7 open models, targeted faults drop from 74.0% to 1.7% wrong with no loss on fault-free tasks; three frontier models fall from 70.2% to 0.5%
- A warning-only variant still leaves 70.1% wrong — enforcement is the key
- On new fault types the harness alone cuts errors from 61.1% to 24.5%, by abstaining rather than recovering
More from coding & agent
- Harvard Paper: Coding Agents Re-read 94.8% of Tokens, Pushing Energy Use 25x Higher — rohanpaul_ai · 2026-11-18
- 1,000+ subagents, 100 hours: GPT-6.2 Astra rebuilds all of Singapore at 1:1 scale — gabrielchua · 2026-11-18
- FastMCP 4.2 ships Skills over MCP — drag a skill folder, get an MCP server instantly — HanchungLee · 2026-11-18
- Amazon paper: weak LLM critics can sink search loops, ideal gain only 0.005 AUC — rohanpaul_ai · 2026-11-18
- OpenAI engineers secretly use Astra for Law to write PRDs and design docs — sherwinwu · 2026-11-18
- START-HERE.md: Mapping Your Vault So Agents Find Files Without Burning Tokens — tomcrawshaw01 · 2026-11-18