Kimi K3.1 Pro trails GPT-5.6-Sol on cyber eval after Moonshot stripped cyber training data
scaling01 · x · 2026-10-24
Moonshot deliberately removed cyber-related data from Kimi K3.1 Pro training, stating that "without data removal, downstream cyber capability would increase substantially" — an intentional capability suppression.
The trade-off shows in the internal xdev-bench-hard (pass@3) eval: GPT-5.6 Sol (codex harness) leads with 100 problems solved and a 77.8 average success rate; an unreduced Kimi variant hits 90.5/71.4, while the shipped K3.1 Pro manages 81.0/64.5, clearly behind. DeepSeek-V4.1-Flash scores 85.7, GLM-5.3 57.1, and the previous Kimi K3 just 33.3.
Notably, K3.1 Pro's successful trajectories use far fewer tokens (159.8M/546K) than the unreduced variant (216.4M/499K) or GLM-5.3 (381.1M/3.6M).
More from Models
- Kimi K3.1 Pro vs GPT-6 Astra vs Opus 5.5: Building Star Trek's Enterprise D in Minecraft — Angaisb_ · 2026-10-24
- Kimi 3.1 Pro priced like Argon but scores like DeepSeek V4.1 Flash on meme 'Runescape Bench' — scaling01 · 2026-10-24
- Jon Barron details his workflow: Gemini runs experiments, drafts the paper, then rewrites in his style — jon_barron · 2026-10-24
- Users praise Gem Argon for cultural fluency: a frontier model that roots for you — sebkrier · 2026-10-24
- User says Opus silently downgraded to 4.8 mid-task, then told him to talk to people — natesiggard · 2026-10-24
- Suspicious new GitHub accounts reference Qwen4 in Alibaba T-Head repo PRs — brainExploded99 · 2026-10-24