EvalSeal v1.5.0: open-source reproducibility receipts for LLM evals
Fit_Fortune953 · reddit · 2026-09-21
A developer released EvalSeal v1.5.0, an open-source tool that makes LLM eval results trustworthy by running cases multiple times, measuring per-case instability, capturing provenance, and sealing results into a tamper-evident ledger.
Features
- Per-case verdict strips and evaluator fingerprinting
- Signed ledger heads, CI gates, drift comparison via evalseal diff
- Deterministic HTML receipts attachable to PRs; replayable examples without API keys
Key finding: with an LLM judge, 5 of 20 borderline cases flipped verdicts across repeated runs; numeric answer matching on 40 GSM8K cases flipped 0. Same model family — the instability came from the evaluator, not the target model.
Related event: EvalSeal Open-Sourced to Bring Trust Receipts to LLM Evaluations(2 posts)→
More from coding & agent
- Addy Osmani shows Claude Code can auto-evaluate whether a plugin improves answers — addyosmani · 2026-09-21
- Devs say avoiding cache misses could boost Claude Code/Codex effective usage limits 10-20% — chaseleantj · 2026-09-21
- Solo dev's open-source AI workflow platform hits 38 deployments, now faces maintenance questions — Feathered-Beast · 2026-09-21
- Where should agent action authorization live? AI support teams debate refund safety policies — witty_queen123 · 2026-09-21
- Two years from learning Claude Code to $35k freelance income and a failed first startup — PratikKadam_ · 2026-09-21
- ostris ai-toolkit ships Qwen-Image-2.1 LoRA training support, no community LoRAs yet — reeight · 2026-09-21