"Benchmarks rule out the bad models — the good ones I test on my own work"
ChemicalExcellent463 · reddit · 2026-11-10
A developer shares his model evaluation methodology: benchmarks are a floor, not a ranking — past a certain score they're all saturated, so they only rule out bad models. The good ones get tested on real work:
- Doing: hand the model another agent's plan and see if it finishes without stalling or freezing at the first problem — across code, search, and writing.
- Checking: whether it finds more bugs, or deeper ones, than the previous model caught. Doing and checking have separate bars, and both keep rising.
He still screens every new model for old habits: lots of talk, little action; holes in logic; misreading the ask and running hard the wrong way.
More from Models
- Google Gemini rolls out Memory summary to view, edit, and delete everything it remembers about you — gaganghotra_ · 2026-11-10
- LMArena launches Agent Arena with a Pareto frontier for agents — arena · 2026-11-10
- HuggingFace decision models are nearly all Qwen-based — ivan_bezdomny · 2026-11-10
- Kaggle Veteran: AI Benchmarks Need Private-Leaderboard-Style Hidden Test Sets — MeganRisdal · 2026-11-10
- Six free open-source models, one per job: from everyday chat to company servers — alex_verem · 2026-11-10
- Grok API text outputs now watermarked by default across 7 models for EU AI Act — thione · 2026-11-10