Kimi K3 looks strong on benchmarks, but a real coding test says reliability still lags
Cole Medin · youtube · 2026-07-24
A YouTube creator benchmarks Kimi K3 against Opus 4.8 and Kimi K2.7 on real engineering tasks and argues the public hype overstates what the model can actually do.
Key takeaways from the video:
- Kimi K3 looks extremely strong on benchmarks and may appear close to or ahead of top closed models.
- In practical engineering work, it still shows reliability failure modes that public leaderboards don’t capture.
- On simple tasks, K3 can roughly tie Opus; on more complex tasks, Opus pulls ahead.
- The creator says open-weight models are still valuable as the workhorse in a mixed-model workflow.
- The video includes an open-source benchmark repo, rubric, prompts, and harness so others can reproduce the tests.
A notable result cited in the video is a failure-rate gap of 8% vs. 36%, used to argue that benchmark design matters more than leaderboard headlines.
More from Models
- DeepSeek cut KV cache per token from 389KB to 890B in two years as open-closed gap debate rages — teortaxesTex · 2026-10-27
- GPT-6 Astra controls dexterous hands zero-shot at just 5.2% success in 500 trials — stingning · 2026-10-27
- Unverified: fable 5.5 speculated at 4-5T parameters to outscale 1-3T Opus — haider1 · 2026-10-27
- Kimi K3 Is 'Open' Until You Succeed: License Caps Free Use at $20M Revenue — alex_verem · 2026-10-27
- Screenshot shows 'GPT-6 Astra Ultra' in ChatGPT model picker — unverified leak or joke — justalexoki · 2026-10-27
- DeepSeek Safety Eval Finds "Over-Attribution": Model Mistakes Ordinary Tasks for Tests — Much_Boysenberry9124 · 2026-10-27