Attest: record agent runs once, replay in CI with zero model calls
tom_mathews · reddit · 2026-11-15
The author released Attest, an Apache-2.0 testing framework for LLM agents (Go engine, Python/TypeScript SDKs) that records every model call behind a local proxy, then replays those recordings in CI — 0 live model calls, $0.00. Deterministic checks (schema, cost/latency limits, tool ordering, simulated-backend effects) run without a model; similarity checks use a bundled ONNX MiniLM; judges point at any OpenAI-compatible local server. Malformed judge replies are "not evaluated", never a pass. Version 1.1 fixes four ways 1.0 could report a failing assertion as passing. Most of the code was written by Claude Code and Codex from the author's plans.
More from coding & agent
- The T.R.U.S.T. check: deciding which jobs you can safely hand to your AI bots — bfrench · 2026-11-15
- Read-only MCP server lets Claude Code write regression tests for agents — tom_mathews · 2026-11-15
- Bot audit experiment: triage plus deep-read cuts input 64% while staying stable — bfrench · 2026-11-15
- How do you track all the AI agents running inside your company? — CupRare814 · 2026-11-15
- Developer builds VR island adventure game with Claude Code writing the C# — Illustrious-Lynx3673 · 2026-11-15
- Building a workspace where agents, not humans, are the primary users — Pristine-Student4702 · 2026-11-15