Papr hits 96.0% on LongMemEval-S with official judge, using a third of rivals' context
remoteinspace · reddit · 2026-11-15
Papr reports 96.0% on all 500 LongMemEval-S questions using the official gpt-4o judge prompt, beating Chronos (95.6%), Hindsight (94.6% at 43.6k tokens/question) and gbrain (90.6%). Answers, judge labels and code are public and reproducible via the official script.
Key findings
- Retrieval recall is near saturated: evidence sessions land in top-5 for 460/470 answerable questions (97.9%); remaining errors come from the reader, not retrieval.
- Top-20 vs top-5 is a trade, not a free win: 11 gained, 8 lost (net +3); temporal reasoning went 123→130/133; abstention flat.
- A small reader gets most of the way: gpt-5-mini reaches 94.4% on the same retrieval; top-5 config hits 95.4% at just 16.5k tokens/question — about a third of Hindsight's context.
Caveats
- Self-reported; run variance ±3 questions, so 96.0 vs 95.6 is a tie; top-20 was chosen after reviewing top-5 misses, so 96.0% is somewhat tuned on this set.
- Uses open-source Papr-Embed 0.6B embeddings and a gemini flash 2.5 reranker; no question-type labels used.
More from Research
- Pinterest swaps clustering pipeline for a fine-tuned 4B LLM, lifts email CTR 1.1% — nilukush · 2026-11-15
- Curated 10-resource path to learn LLM distributed training from scratch — East-Muffin-6472 · 2026-11-15
- Broad/MGH preprint models inherited variation effects on disease trajectories using longitudinal EHRs — anshulkundaje · 2026-11-15
- Llama 405B Base Model Natively Mimics o1's Chain-of-Thought from 128 Pages of Context — Sauers_ · 2026-11-15
- Magic claims >10x more efficient pretraining, matching DeepSeek V4 Base for ~$0.5M — oran_ge · 2026-11-15
- OpenDS-World benchmark: best of 12 frontier models solves only 39% of real-web deep research tasks — jiqizhixin · 2026-11-15