New Benchmark Tests Agents on Formal Proofs
OfirPress · x · 2026-07-09
Balaji Rao and colleagues introduced s2n-bignum-bench, a benchmark evaluating agents' ability to write proofs in HOL Light for cryptographic assembly code from Amazon's s2n-bignum repository. Codex 5.3 xhigh currently scores just 6.3%, indicating substantial room for improvement on this benchmark.
More from coding & agent
- Alchemy's Lambda Web Functions run plain Node web servers on Lambda, no OpenNext needed — samgoodwin89 · 2026-10-27
- Alchemy Domain brings domain-driven design and story testing to Effect backends — samgoodwin89 · 2026-10-27
- Alchemy makes all ~2,850 AWS bindings work from any cloud, not just S3/Lambda/DynamoDB — samgoodwin89 · 2026-10-27
- Production AI agent scored 95% robustness at baseline, 37% under adversarial attack — njyx · 2026-10-27
- Chaining Grok Bot and Codex to detect and mass-block AI reply accounts — vista8 · 2026-10-27
- huggingface_hub 2.3.0 cuts 1M-file listings to 20s, adds live Bucket following — lhoestq · 2026-10-27