Researchers spar over code perplexity evals as Kimi-K3 tops all 10 codebases
eliebakouch · x · 2026-10-24
- HF researcher eliebakouch questions the value of code-perplexity benchmarks in a Baseten eval discussion: a code-only trained model beating larger all-domain models on perplexity is expected; what matters is performance on more realistic code benchmarks, and he'd like pass@k across coding benchmarks.
- The table he shared ("another perplexity eval") shows mean bits-per-byte across 10 codebases: Kimi-K3 ranks first on all 10 (tied 0.127 with Mirendil-Base on json-render), leading Mirendil-Base, Nemotron-550B, GLM-5.3 and DSV4-Pro.
Related event: Researchers debate credibility of coding benchmarks as pass@k can be gamed(4 posts)→
More from Models
- OpenAI red team finds critical Brotli RCE (CVE-2026-11149) hitting Node.js, Bun, Envoy — moyix · 2026-10-24
- Kimi K3.1 falls short of benchmarks in hands-on testing, says KOL — DeryaTR_ · 2026-10-24
- Kimi K3.1 Pro tops agentic coding benchmarks but may lag on AA-3D — ArtificialAnlys · 2026-10-24
- ValsAI Index Finds Fireworks AI Beats DeepSeek and Moonshot's Own APIs on Agentic Tasks — sophiamyang · 2026-10-24
- TensorSharp hits 5.5× llama.cpp prefill throughput on DeepSeek V4.1 Flash across five A40s — fuzhongkai · 2026-10-24
- Kimi weights rumored to land Nov 4: would you self-host a 2.8T MoE at ~$2/$10 API pricing? — AIFlow_ML · 2026-10-24