New Benchmark Tests Agents on Formal Proofs
OfirPress · x · 2026-07-09
Balaji Rao and colleagues introduced s2n-bignum-bench, a benchmark evaluating agents' ability to write proofs in HOL Light for cryptographic assembly code from Amazon's s2n-bignum repository. Codex 5.3 xhigh currently scores just 6.3%, indicating substantial room for improvement on this benchmark.
More from coding & agent
- A 9B Ollama agent can run a fully local DJ radio with tools, memory, and TTS — pinku1 · 2026-07-27
- Bugbot rejects an MCP permission flag because it would break path-scoped isolation — zeeg · 2026-07-27
- One GPT-5.6 agent is guarding a Blink security system while another makes a parody rap album — repligate · 2026-07-27
- An agent got unblocked by reusing a logged-in browser, not stealth tricks — armanidev_ · 2026-07-27
- Paper argues graph topology can become the core operating system for AI agents — theomitsa · 2026-07-27
- Claude Code desktop adds UI markup feedback for smoother visual editing — EricBuess · 2026-07-27