lm-eval-ledger: an open-source harness to browse and compare per-question model answers
jayminban · reddit · 2026-09-07
Developer jayminban released lm-eval-ledger, an open-source LLM benchmark harness fixing the pain that existing tools only hand you headline scores while answers are buried in JSONL/Parquet dumps.
- Records and visualizes per-question data: system prompt, raw generation, extracted answer, ground truth, stop reason, generation length; per-benchmark accuracy, tok/s, runtime, sample count, and no-answer count
- Supports pairwise comparison of two models on the same questions, plus always-wrong/always-right question filters across runs
- Everything lands in a single SQLite DB behind a Flask web app; YAML-driven so one command runs N models × M tasks
- Verified on vLLM, SGLang, HF, and llama.cpp server backends (Linux and Windows); all demo runs done on a single RTX 5090
Live demo on Hugging Face Spaces, code on GitHub, feedback and PRs welcome.
More from coding & agent
- Dev says he has his Jarvis: no longer reads agent replies at all — BLUECOW009 · 2026-09-07
- Median OpenAI researcher spent $0/day on coding agents in February, post reveals — zacharynado · 2026-09-07
- Switching an LLM pipeline to structured outputs killed a months-long silent bug — ClickOk5811 · 2026-09-07
- Replotting coding-agent costs by API price flips the picture, and OpenAI devs' median daily spend was near $0 — eliebakouch · 2026-09-07
- Building a Python interpreter in just 1024 bytes of C code — AustinZHenley · 2026-09-07
- Doubao Agent Wows in Live Demos: One Vague Prompt, Full PPT with Context from Feishu — AGI Hunt · 2026-09-07