TerminalBench 4.0 questioned: Kimi K3.1's top score may leak hidden tests, per new critique
teortaxesTex · x · 2026-11-03
A critique argues Kimi K3.1's TerminalBench lead likely comes from knowledge of hidden test sets: it excels on the hardest tasks where all frontier models struggle, but performance collapses when tasks are slightly modified. Reposter teortaxesTex doubts TB 4.0's reliability overall but notes Kimi looks strong on several "definitely-not-benchmaxxed" new evals.
Related event: Kimi K3.1 Accused of Benchmark Contamination on TerminalBench(2 posts)→
More from Models
- JFPuget's team takes 3rd on both Kaggle ARC-AGI competitions ending the same day — JFPuget · 2026-11-04
- Tencent's WeChat Vision team open-sources WeVisOne-VL, a unified 2B/4B perception VLM — bdsqlsz · 2026-11-04
- DeepMind-Stanford paper: a red heart in your prompt flips code from 0% to 100% recursive — rohanpaul_ai · 2026-11-04
- A Red Heart Can Steer Code: DeepMind-Stanford Paper Probes Visual Meaning in VLMs — rohanpaul_ai · 2026-11-04
- DeepSeek open-sources DeepInfer inference engine delivering 26,000+ tokens/sec per GPU — bycloud · 2026-11-04
- DeepMind RL-trained Gemini doubles correct psychiatric differential diagnoses in blind study — SRSchmidgall · 2026-11-04