Attest 1.1: replay-based agent testing gates, mutants expose 0/29 catch rate for output-only tests
tom_mathews · reddit · 2026-11-15
Attest 1.1 (Apache-2.0) is a testing framework for LLM agents built around a real failure mode: a double refund where every output assertion passes. attest capture records OpenAI/Anthropic calls plus OTLP traces through a loopback proxy, redacting secrets and templating volatile values; behavior contracts like toolcardinality("issuerefund", atmost=1 per orderid) and effect contracts on a simulated payments API gate PRs with 0 live model calls and $0 cost. Mutation testing exposed the gap: output-only tests caught 0 of 29 mutants; contracts caught 10. The 1.1 audit logged 73 findings, fixing four ways a failing assertion could pass (stdio pipe verdict shifting, timed-out response reuse, concurrent Vitest swaps, empty pytest-xdist reports). Most code was written by Claude Code and Codex from milestone plans the author reviewed. Design partners wanted.
More from coding & agent
- AI Evals Guide: Generic Metrics Fail, Top Teams Rely on Error Analysis — PawelHuryn · 2026-11-16
- 45 agent skills from shipping three live products, open-sourced on GitHub — PawelHuryn · 2026-11-16
- PM field guide: building the same CRM in Lovable, AI Studio, and Claude — PawelHuryn · 2026-11-16
- Build a multi-tenant SaaS in 3-4 hours with Claude Code, Clerk, and Supabase — PawelHuryn · 2026-11-16
- Anthropic's 11 open PM roles pay $305K-$460K — none require a CS degree — PawelHuryn · 2026-11-16
- Formal methods tip: have your coding agent write specs and focus on invariants — blaizedsouza · 2026-11-16