EACL 2027 Paper Finds up to 74% Logical Error Rate in LLM Code Across 10 Models
ponguru · x · 2026-11-17
A paper by Adobe Research (India) and IIIT Hyderabad, Measuring the Mistakes: Evaluating Code Hallucinations in LLMs, is accepted to the EACL 2027 Main Conference.
- Evaluates 10 LLMs on generation, debugging, refactoring, and editing tasks across Python, Java, and C++
- Combines hallucination metrics, functional correctness, and code quality into a Code Reliability Index leaderboard
- Reports logical error rates up to 74%; typical hallucinations include inventing non-existent or overly complex packages (e.g. itertool instead of itertools) and turning a simple indexing fix into convoluted infinite-loop logic that changes program behavior
More from Research
- AI agents rewrote 30,000 articles of French law; blind audits found 54% fewer legal inconsistencies — IgorCarron · 2026-11-17
- COrigami: open-source pipeline turns prompts into flat-foldable origami with Gemini and RL — TZahavy · 2026-11-17
- How rollout staleness hurts on-policy distillation, and how to build a faster pipeline — burkov · 2026-11-17
- Apple robotics paper: 91% success in unfamiliar homes using synthetic sim data — FuSheng_0306 · 2026-11-17
- Hierarchos-Native v0.2.0: Rust+Vulkan LLM training adds RDNA3 WMMA and native multimodal training — PhysicsDisastrous462 · 2026-11-17
- Cache engineering beats context engineering: Tsinghua's AKV cuts agent inference cost 27.7% — 机器之心 · 2026-11-17