Comparing 4 Generations of Kimi Agents on Bug Fixing
qubridInc · reddit · 2026-07-16
Comparing 4 Generations of Kimi on the Same Agentic Bug-Fixing Task
The author assigned the same buggy Python repository to K2 Thinking / K2.5 / K2.6 / K2.7 Code sequentially. The models were required to strictly follow the STEP/TOOL/RESULT protocol to compare their performance in "long-chain agentic coding."
Main Conclusions
- Newer models hallucinate test failures less: K2 Thinking misidentifies non-existent KeyErrors as real bugs, K2.5 still fabricates failures, K2.6 is mostly accurate, and K2.7 Code achieved zero fabricated test results.
- Real bug-fixing capability doesn't improve perfectly linearly:
- K2 Thinking found 3/3 bugs but got stuck in an "infinite loop" of misjudgments during simulated tests.
- K2.5, K2.6, and K2.7 all ultimately produced runnable code.
- Execution efficiency improved significantly: K2 Thinking took 386 seconds, K2.5 took 135 seconds, K2.6 took 106 seconds, and K2.7 took 63 seconds. The author noted a cost drop from $0.060 down to $0.017 per run.
Interesting Observations
- K2.6 wrote an undo test to delete SKUs first, then fixed the design based on the failure reason—almost like "accidental TDD."
- K2.7 discovered a "silent bug not covered by tests" but didn't proactively report it, showing a stronger tendency towards strict test constraints.
- The author concludes: If K3 truly targets long-chain agents as rumored, the most crucial aspect to watch in this comparison is whether it can retain K2.7's zero-hallucination discipline while restoring the sensitivity to track issues beyond tests seen in K2 Thinking.
More from Models
- Bug Hunt Bench ranks frontier coding models on 105 planted real-repo bugs — PawelHuryn · 2026-09-11
- GPT-6 Astra beats Factorio with enemies in 44 in-game hours at ~$4,500 API cost — liminal_bardo · 2026-09-11
- 105 hidden bugs, 2 repos: DeepSeek V4.1 Flash fixes 24 at $1.80 vs Opus 5's 27 at $51.33 — ChartsJournalX · 2026-09-11
- awesome-llm-leaderboards: an open-source directory of LLM leaderboards, pricing tables, comparison tools — Last_Establishment_1 · 2026-09-11
- Anthropic claims it works to keep eval environments unidentifiable to models — MaxKannen · 2026-09-11
- Nex N2.5 Pro released on Hugging Face with 407GB of weights — jinnyjuice · 2026-09-11