Kimi K3 leads a 69-task coding benchmark as ML workflow mistakes still trip models
mariofilhoml · x · 2026-08-04
The post argues that coding LLMs are now around “Competitions Master” level on this kind of ML work, but still far from effortless reliability.
The quoted thread describes an experiment using Zoomcamp sales-forecasting data to test a feature-engineering assistant:
- the model tried to transform data before splitting it, creating leakage risk;
- when asked to build 30 daily prediction models from historical data only, it still proposed rolling windows that would leak future information;
- the output repeatedly showed that even strong coding models can stumble on basic ML workflow discipline.
The attached chart compares several models on a task set, with Kimi K3 leading at 49/69 tasks passed, followed by Claude Opus 4.8 at 43/69 and GPT-5.6 Sol xhig at 41/69. The broader message is that these systems are improving, but they still need strong guardrails for real ML engineering.
Related event: LLMs hit master-level coding but lack stability(2 posts)→
More from coding & agent
- Paper maps 41 agent failure modes to model, harness, tools and environment — omarsar0 · 2026-08-04
- Stanford’s CS239A course on self-improving AI agents is now on YouTube — mariofilhoml · 2026-08-04
- Cline says open-weight models work better when the harness lets them verify more — cpaik · 2026-08-04
- WorkOS schedules Agent Night in San Francisco with context-graph talks and live demos — davidcrawshaw · 2026-08-04
- A curated reading list for DeltaNet, FlashKDA, vLLM serving and MoE — austinvhuang · 2026-08-04
- Yann LeCun says strong code-generation systems go beyond plain autoregressive LLMs — ylecun · 2026-08-04