A usage benchmark for AI models: 6 tests you can run on your own work in 2 weeks
shashib · x · 2026-11-16
Shashi Bellamkonda proposes a usage benchmark that replaces technical leaderboards with your own work: run 5 recurring tasks through each model for two weeks and score six observable metrics.
- Editing time: minutes from first output to something you'd send
- Invented facts: check named facts across 10 outputs, count errors
- Rule retention: give 10 written rules, see how many survive into the final third of a long draft
- Pushback: plant one error in the prompt; pass if the model flags it
- Freshness: ask about a page that went live in the past 24 hours
- Return rate: on day 14, check browser history for which tool you actually reopened
Scores vary by reader since tasks differ — hence run it yourself. Tools needed: a spreadsheet and a timer.
More from Research
- Chollet: reasoning chain generation is search, not interpolation — teortaxesTex · 2026-11-16
- Factorised output heads let tiny LMs use a 131k vocab at equal training FLOPs — BLCNYY · 2026-11-16
- Terence Tao: math has moved from proof scarcity to proof abundance — haider1 · 2026-11-16
- Wrong AI advice hurts doctors' judgment more than right advice helps: study of 16,641 decisions — ValerioCapraro · 2026-11-16
- Free online book 'The Science of Computing' links PDEs, numerics, and cache performance — burkov · 2026-11-16
- Controlled speech experiments: joint training from scratch always beats separate pretraining — giffmana · 2026-11-16