Informal verification: using System One models as proof checkers to build epistemic trust in agents
LauraRuis · x · 2026-11-22
Nikolas Goebel's essay, shared by Laura Ruis, argues that letting agents modify complex systems requires epistemic trust—believing the system works the way you think it does—not just an empirical track record.
- Classic example: a machine that appears to sort marbles by color may actually sort by size, equivalent under normal operation until a large blue marble exposes the misalignment. Incident-free weeks alone don't reveal this.
- Verification is fundamentally an exercise in building trust; formal verification is now practical but not necessary.
- System One models can act as informal proof checkers, accepting what is 'intuitively correct' while forcing authors to spell things out at a higher level of detail—a complementary tool in agentic software engineering workflows.
More from coding & agent
- Gemini 4 Neon tops Bug Hunt Bench; Antigravity vs AFK Pilot vs T3 Code compared — PawelHuryn · 2026-11-22
- NotebookLM + Google Antigravity: 4 free prompts to compress hours of research into minutes — aftahi_ai · 2026-11-22
- Dev warns: revoke API keys after feeding them to models, or use hourly temp credentials — lipeng0820 · 2026-11-22
- Lauren Tan's team ships 11,000 PRs a month with ~0.5% revert rate using agents — aftahi_ai · 2026-11-22
- Rudrank shares his self-built AI orchestrator project — rudrank · 2026-11-22
- Dev shows ASC CLI maintained daily by Opus 5.5 in 'For Agents, By Agents' talk — rudrank · 2026-11-22