Building adebench: scoring what your agent's memory actually delivers, 93.7/100
Soft-Lie-434 · reddit · 2026-09-12
A developer built adebench (MIT, zero dependencies) to evaluate his personal assistant's memory service (working/episodic/semantic memory, knowledge graph, entity cards exposed via MCP to Claude, Codex and a voice client).
Key idea: score the text the client actually receives through a "door", not retrieval hits — a fact retrieved but landing at character 3,000 doesn't exist for the model. Measuring the agent door revealed agents got less than the voice client (17/24 vs 21/24); one fix brought them to 22/24.
Eight scored sections (0-100): door (25), cards (15), updates (10), time (10), livestate (10), abstention (10), filesearch (10), graph (10), plus report-only lifecycle health and latency-per-door sections.
v0.1 mistakes worth stealing from: HTTP 500 scored as perfect abstention; an empty graph scored 10/10; '8766' matched '18766'; same-minute runs overwrote each other. Now everything is PASS/FAIL/ERROR/SKIP, errors never earn points, and 37 CI tests guard each failure mode.
His Brain scores 93.7/100 on his 24-question golden set — comparable only over time, not across systems. Other memory systems plug in via a 25-method adapter, with a two-second synthetic example included.
More from coding & agent
- Real2sim first, then real: user reproduces Astra controlling a robot arm to paint — ZeYanjie · 2026-09-12
- Eight years of logging electronic parts in .txt files now feeds AI project context — debreuil · 2026-09-12
- Worktrunk, a Rust CLI for Git Worktrees Built for Parallel AI Agents, Hits 7k Stars — max-sixty · 2026-09-12
- Claude-Red: Open-Source Red-Team Skill Library for Claude Hits 3.3k Stars — SnailSploit · 2026-09-12
- Codex Pet Update Adds Popup Replies with Voice and Text Input, No Client Needed — xiaohu · 2026-09-12
- BossConsole: open-source desktop harness to run AI coding agents — Roger_M_Taylor · 2026-09-12