A personal benchmark system to track frontier AI progress and cheap-model ceilings
bendee983 · x · 2026-11-10
bendee983 shares a practical methodology (reacting to levie's advice to recheck AI every 3-6 months):
- Keep a set of tasks that current models can almost do without scaffolding; re-test each new model release to gauge frontier capability.
- When a task saturates, replace it with a harder version for frontier models.
- Don't discard saturated benchmarks — run them on cheap non-frontier models to see how far you can push them. He notes Flash-level open LLMs under $1/M output tokens (e.g. DeepSeek-v4.1 Flash, GLM-5.3 Flash) can do surprisingly much.
Core idea: one near-saturation benchmark suite measures both the frontier edge and cheap models' at-scale usability.
More from AGI Musings
- Toby Ord: monitoring the research-spend curve could signal an AI explosion — tobyordoxford · 2026-11-10
- Toby Ord and Tom Davidson debate whether the AI research-spend curve breaks — tobyordoxford · 2026-11-10
- billion-context: open-source gene expression tokenizer hits 92.4% accuracy on cell-type classification — udmrzn · 2026-11-10
- Researcher: Agents Are Compounding Gains by Accelerating Scaling-Law Discoveries — _arohan_ · 2026-11-10
- Enterprise AI willingness-to-pay could hit 8 figures if you solve their internal workflows — gabriel1 · 2026-11-10
- Toby Ord: AI revenue may scale exponentially with intelligence, but attribution unclear — tobyordoxford · 2026-11-10