antirez: stellar cybersec benchmarks aside, wait for real user testing before trusting the best scores
antirez · x · 2026-09-10
antirez continued his take on the new DeepSeek model: benchmarks in areas like cybersecurity look terrific — a huge use case — but other benchmarks align more with past performances. He advises waiting for real user testing before trusting the best benchmark scores as a general proxy for capability.
More from Models
- 'Gemini 3.8 Flash' demo claims task completion with self-correction in 3 turns — Artistic_Solution117 · 2026-09-10
- "Alien architecture" model design stuns, blogger suggests layering recurrent depth on top — scaling01 · 2026-09-10
- Users Petition OpenAI for $400-$600 Heavy Builder Tier as $200 Plan Runs Dry in 48 Hours — dragonwarrior_1 · 2026-09-10
- Leak: 'SpaceXAI' working to bring Grok Bots into XChat for in-conversation tagging — nima_owji · 2026-09-10
- Chinese model's 74.2 score under fire: best of 8 eval variants, maxed thinking budget, ~2.5x cost — teortaxesTex · 2026-09-10
- Unitree fully open-sources UnifoLM-WLA-1.0, a 6B humanoid robot foundation model — teortaxesTex · 2026-09-10