Syntax Highlighting Benchmark Using Shiki as Ground Truth Draws Accuracy Dispute
DoctorGester · x · 2026-09-09
- shuding explained his benchmark methodology: comparing span types against Shiki (real AST-based, near gold standard), reporting weighted correctness across GitHub's top 25 popular languages.
- DoctorGester pushed back, calling results misleading: dozens of incorrect highlights are easy to find, with roughly 50% of threejs.min.js marked wrong and confusing spans in other files.
More from Models
- Open Models Match GPT-6 on MedMCQA but Trail 17 Points on Hard Clinical Cases — keunwoochoi · 2026-10-27
- Health-Focused Benchmark Reranks LLMs: Muse Spark Open Jumps from #9 to #1 — keunwoochoi · 2026-10-27
- DeepSeek cut KV cache per token from 389KB to 890B in two years as open-closed gap debate rages — teortaxesTex · 2026-10-27
- GPT-6 Astra controls dexterous hands zero-shot at just 5.2% success in 500 trials — stingning · 2026-10-27
- Unverified: fable 5.5 speculated at 4-5T parameters to outscale 1-3T Opus — haider1 · 2026-10-27
- Kimi K3 Is 'Open' Until You Succeed: License Caps Free Use at $20M Revenue — alex_verem · 2026-10-27