Notion's real-world benchmark: open models hit 94-98% quality, cost as low as $0.02 per task
ivanhzhao · x · 2026-08-14
Notion's research team launched the Notion Knowledge Board, evaluating 15 models on real production traffic for real-world knowledge work, scored by an ensemble judge built with researchers from Anthropic, OpenAI, and Google.
Key findings:
- Quality is close across all models: 94-98%, including open source.
- Cost varies widely: $0.02 to $0.87 per task.
- For most everyday work, cheaper models are now good enough; choosing the right model can cut costs without losing much quality.
Related event: Notion Launches Real-Traffic Model Evaluation Board(4 posts)→
More from Research
- 200 chess games suggest in-context learning doesn't work yet, experimenter says — akbirthko · 2026-10-16
- Mosaic: multi-session agentic RL with hierarchical advantage estimation — LARK-Lab · 2026-10-16
- Meituan LongCat's WorldCast benchmark: GPT-Image-2.5 tops at 72.8, open-source lags — meituan-longcat · 2026-10-16
- Recursive Environment Scaling tanks six models' success rates by 57–77 points — Lingxiao Jing · 2026-10-16
- Harvard's MPT: decentralized agent swarms learn self-orchestration via bidding — Harvard · 2026-10-16
- MIT Biology and Schwarzman College hiring tenure-track professor in computational biology with generative AI focus — anshulkundaje · 2026-10-16