Retrieval Unchanged, But 32 of 500 LongMemEval Results Flipped
przemarzec · reddit · 2026-08-21
Comparing Engrava 0.6.0 and 0.5.0 on LongMemEval revealed that while retrieval results were identical for 457 questions, 32 outcomes flipped (18 down, 14 up). This proves the score variance occurred post-retrieval in the reader or judge stage, not in the memory layer itself. The post advises diffing retrieved context IDs, not just final scores, when benchmarking memory layers.
More from Research
- Eris theorem proving environment updates with direct manipulation — round · 2026-08-21
- Nature Paper: Consumer AI shifts from info tool to healthcare pathway control — EricTopol · 2026-08-21
- DeepMind uses AlphaEvolve to improve matrix multiplication bound — rohanpaul_ai · 2026-08-21
- Hydra-0 unifies action language for physics simulation and robot control — YunzhuLiYZ · 2026-08-21
- David Baker's Team Publishes on Accelerating AI Protein Design Validation Loop — AllThingsApx · 2026-08-21
- CP-MoE: Freeze the Base, Tune Just 1.5% Params, No Forgetting — flosalim · 2026-08-21