SurrealDB: Agent memory benchmark headlines run 20+ points above strict scores
mattturck · x · 2026-10-16
SurrealDB launched Agent Memory and simultaneously pulled back the curtain on agent-memory benchmarks. Its own BEAM score at 1M tokens is 77.2% on the headline metric but only 57.1% counting full marks — a 20-point gap.
The team read the test harnesses, prompts, judge settings and per-question outputs behind published BEAM and LoCoMo scores, arguing a score is a "recipe": which model answers, which judges, what prompts they get, and how rubric items are scored all change the number. The chart shows similar gaps across vendors: Exabase 75.0%→48.0% strict, Hindsight 73.9%→48.4%, Mem0 64.1%→40.1%, Honcho 63.1%→41.4%, while HydraDB lists 82.0% with no strict count.
SurrealDB stresses this isn't a ranking — systems used different answering models, judges and scoring rules — but says headline scores are hard to interpret without full harness disclosure.
Related event: SurrealDB Launches Agent Memory, Discloses Stricter BEAM Score Gap(5 posts)→
More from coding & agent
- Vibe Coding a Data Pipeline Is Easy; Running It at Billions of Rows Is Not — Zachly · 2026-10-16
- How Claude Code Review works at GA: admin setup, auto reviews on every new PR — ClaudeDevs · 2026-10-16
- Anthropic offers eligible Claude Enterprise orgs at least $10,000 in Code Review credits — ClaudeDevs · 2026-10-16
- Anthropic makes Code Review generally available for Claude Team and Enterprise plans — ClaudeDevs · 2026-10-16
- Supabase CLI v2.119.0 Drops Docker, Gives Each Coding Agent Its Own Local Project — dshukertjr · 2026-10-16
- Musk amplifies user claim that Grok's computer use is fast enough to skip a Codex subscription — elonmusk · 2026-10-16