Notion Launches Knowledge Board: Evaluating LLMs on Real-World Traffic Instead of Benchmarks
ivanhzhao · x · 2026-08-14
Notion has introduced the Knowledge Board, a model evaluation dashboard. Unlike traditional benchmarks using fixed tasks, it uses a small, random, and anonymized sample of live platform traffic, split evenly across models to handle actual knowledge work.
Developed in collaboration with researchers from Anthropic and OpenAI, the system uses an ensemble judge to measure resolution rate, time, and cost. Because many model differences fall within confidence intervals, Notion deliberately avoids creating a traditional leaderboard, instead plotting quality against cost to help users find the right model for their specific needs.
Related event: Notion Launches Real-Traffic Model Evaluation Board(4 posts)→
More from Models
- 200 chess games suggest in-context learning doesn't work yet, experimenter says — akbirthko · 2026-10-16
- OpenAI's internal model shows real math "research taste," experts say its proofs go beyond prior literature — cong_ml · 2026-10-16
- User catches Opus spinning up throwaway zz- debug scripts, echoing the METR OAI-HuggingFace incident — holdenmatt · 2026-10-16
- OpenAI drops 722 math papers tied to unreleased model, mathematicians demand receipts — thursdai_pod · 2026-10-16
- Fable 5/5.1 models show a striking drive toward record keeping, observers report — repligate · 2026-10-16
- Google's Gemini 4 Argon performs noticeably worse in AI Mode than in AI Studio — gaganghotra_ · 2026-10-16