Kimi K3.1 Pro ships nerfed on cyber: internal bench drops from 90.5 to 81.0 after data removal
teortaxesTex · x · 2026-10-24
Discussion of Moonshot AI deliberately reducing Kimi K3.1 Pro's cybersecurity capability, with the internal xdev-bench-hard (pass@3) table.
- A checkpoint without risk-reduction (no mid-training cyber data removal, no cyber RL environment filtering) solves 90.5% of problems; the final K3.1 Pro reaches 81.0%, showing data removal meaningfully cuts cyber capability.
- Context in the same table: GPT-5.6 Sol 100, DeepSeek-V4.1-Flash 85.7, GLM-5.3 57.1, Kimi K3 just 33.3.
- Commentator teortaxesTex notes K3.1 Pro still beats GLM-5.3 and nears V4.1-Flash; without the nerf it would have been close to 5.6 Sol.
- Moonshot adds that stronger general coding and long-horizon debugging still generalize to cyber tasks despite targeted data removal.
More from Models
- Fireworks explains how it made RL practical on Kimi K3 at 2.8T parameters — sophiamyang · 2026-10-24
- Astra finds bug in Anthropic's formal verification example: spec doesn't check output matches input — teortaxesTex · 2026-10-24
- Update: open weights for the Fable-class local model coming November 4th — kimmonismus · 2026-10-24
- Microsoft & HF turn database checks into RL rewards, lifting Qwen3.8-27B agents from 51.7% to 60.78% — SergioPaniego · 2026-10-24
- Frontier-level intelligence goes local: Kimi K1.5 Pro hits 55 on AA index, closing on Claude — kimmonismus · 2026-10-24
- Kimi K3.1 Pro vs GPT-6 Astra vs Opus 5.5: Building Star Trek's Enterprise D in Minecraft — Angaisb_ · 2026-10-24