Notion's real-world benchmark: open models hit 94-98% quality, cost as low as $0.02 per task
ivanhzhao · x · 2026-08-15
Notion's research team launched the Notion Knowledge Board, evaluating 15 models on real production traffic for real-world knowledge work, scored by an ensemble judge built with researchers from Anthropic, OpenAI, and Google.
Key findings:
- Quality is close across all models: 94-98%, including open source.
- Cost varies widely: $0.02 to $0.87 per task.
- For most everyday work, cheaper models are now good enough; choosing the right model can cut costs without losing much quality.
Related event: Notion Launches Real-Traffic Model Evaluation Board(4 posts)→
More from Research
- AI for Science Symposium on 8/27 to feature talks on AI accelerating discovery — _shreya_s · 2026-08-15
- High School Student Uses AI to Identify 1.5 Million Potential Variable Stars in NASA Data — PeterDiamandis · 2026-08-15
- 27B agent Faraday beats Claude Opus 4.8 and GPT-5.5 on research replication via new Replica method — omarsar0 · 2026-08-15
- AI Reprocessing of 80K PDB Structures Reveals 60K+ Protein Conformational Ensembles — anshulkundaje · 2026-08-15
- Neuro-symbolic world models gain support: top ARC-AGI-3 harnesses use this approach — GaryMarcus · 2026-08-15
- CMU PhD on adaptive AI agents: from memory & skills to context adaptation, what's the challenge? — Diyi_Yang · 2026-08-15