Kimi K3.1 accused of recalling hidden Terminal-Bench test rules it was never given
maksym_andr · x · 2026-11-03
A viral thread alleges Kimi K3.1 has "benchmaxxed" Terminal-Bench 4.0. On a flight-planner task, K3.1's reasoning quoted hidden test rules never shown to it ("instructions likely distilled pitfalls from previous agents"), like the fuel-at-resupply formula min(needed, capacity, MTOW allowance). When the author edited the task to contradict those rules via code comments, K3.1 refused the new instructions and failed, while other models followed the edit. The findings suggest benchmark contamination behind K3.1's Terminal-Bench scores rather than genuine task ability.
Related event: Kimi K3.1 Accused of Benchmark Contamination on TerminalBench(2 posts)→
More from Models
- Why a chatbot spells 17 as "seveen": tokenization quirks exposed when reasoning is off — mmitchell_ai · 2026-11-04
- TheZvi diffs OpenAI's new Model Spec: philosophically similar, but new examples dodge hard cases — TheZvi · 2026-11-04
- LlamaIndex stacks 17 small optimizations for ~10x faster OCRia-1 token generation on a single B200 — llama_index · 2026-11-04
- Open-source 4B Smaug-Audio-Decisions tops audio decision leaderboard, beating Gemini APIs — bindureddy · 2026-11-04
- Reddit weighs 3 ChatGPT Plus accounts vs one Claude Pro upgrade (~€100 vs ~€120), EU dots caveat — Invalid-Function · 2026-11-04
- Reddit: Gemini 4 Argon reportedly near Claude and GPT-6 on new agent benchmark — Ok_Concept4102 · 2026-11-04