DeepSeek 1017 gains just 0.6% on CyberGym as Flash tops leaderboard at $0.23 per task
teortaxesTex · x · 2026-10-31
teortaxesTex analyzes Fireworks' third-party evals: DeepSeek 1017 scores 95.36% pass@3 on CyberGym vs 0910's 94.82% — a 0.6% gain within overlapping confidence intervals. The accompanying leaderboard shows DeepSeek V4.1 Flash (1017) on top, costing just $0.23/task at 11m34s versus GLM 5.3 ($2.25) and Kimi K3 ($2.89); ExploitBench hits 96.8% at 3000 steps. He argues distinguishing high pass@1 models requires steep difficulty distributions: under a homogeneous-task assumption, 0910's 88.1% pass@1 should yield 99.8% pass@3, but the reported 94.8% implies failures concentrate on stubborn hard tasks (a first-attempt failure is itself evidence of difficulty). Covers harness settings (temp 1.0, 131k output tokens, 6h cap) and 0-3% unscorable tasks, and asks Fireworks to publish pass@1. Unofficial analysis.
Related event: Rumored DeepSeek Cyber Model Benchmarks Surface Online(4 posts)→
More from Models
- SenseNova-U1.5-8B hits 125 tok/s on M5 Ultra in llama.cpp and spots a $1 receipt error — Binary_orchid · 2026-10-31
- Pundit: Anthropic killing 5-hour limit plus Fable launch makes it unstoppable — iruletheworldmo · 2026-10-31
- ChatGPT Daybreak Blue sessions may train OpenAI models even with training opt-out off — littmath · 2026-10-31
- Claude refuses to draw its own logo over copyright concerns — 141_1337 · 2026-10-31
- AI cracks a 27-year-old pillar of computer science, OpenAI and Anthropic claim — Fcking_Chuck · 2026-10-31
- Perplexity early evals: Gemini 4 Carbon tops WANDR benchmark for long-form research — AravSrinivas · 2026-10-31