Kimi K3 looks strong on benchmarks, but a real coding test says reliability still lags
Cole Medin · youtube · 2026-07-24
A YouTube creator benchmarks Kimi K3 against Opus 4.8 and Kimi K2.7 on real engineering tasks and argues the public hype overstates what the model can actually do.
Key takeaways from the video:
- Kimi K3 looks extremely strong on benchmarks and may appear close to or ahead of top closed models.
- In practical engineering work, it still shows reliability failure modes that public leaderboards don’t capture.
- On simple tasks, K3 can roughly tie Opus; on more complex tasks, Opus pulls ahead.
- The creator says open-weight models are still valuable as the workhorse in a mixed-model workflow.
- The video includes an open-source benchmark repo, rubric, prompts, and harness so others can reproduce the tests.
A notable result cited in the video is a failure-rate gap of 8% vs. 36%, used to argue that benchmark design matters more than leaderboard headlines.
More from Models
- Fern's "context is intelligence" thesis: prediction keeps improving out to 10.2M tokens — bneyshabur · 2026-10-26
- Mirendil launches Fern, a from-scratch pretrained model with up to 10M-token context — bneyshabur · 2026-10-26
- Users spot a dedicated translation UI appearing in Claude — BLCNYY · 2026-10-26
- GrokBot user: flight booking and shopping all blocked, agent can't do anything useful — lucasmeijer · 2026-10-26
- Vending-bench-style eval: Sol 5.6 appears the most aggressive contract-violator — xuanalogue · 2026-10-26
- Fable 5.5 launch rumors persist for third week, but no frontier model drops this month — patience_cave · 2026-10-26