Benchmark reveals LLMs struggle with real-world tasks despite coding prowess

数字生命卡兹克 · wechat · 2026-08-17

This article analyzes several benchmarks designed to test LLMs on real-world tasks, revealing a significant performance gap compared to their success in coding.

Key Findings

E-commerce: In AliExpress's RealReplicaBench (107 real tasks), only Claude Opus 5 passed 60.75%, with most models under 50%. Tasks involve complex, messy data like sourcing, logistics, and disputes.
Product/Lifestyle: Benchmarks like Tencent's E-Bench (Gaming/Music apps) and Xiaohongshu's VibeLifeBench show low average pass rates and poor stability across multiple runs.
Coding vs. Real Work: Models achieve >70% on coding benchmarks like DeepSWE but often fail real-world tasks. The gap is attributed to the unstructured, chaotic nature of real work.

Evaluation Mechanism

Benchmarks simulate full workflows (e.g., using tools, checking email states) with strict grading (one error fails the task), emphasizing results over process.
Specialized agent frameworks (like Accio) improve performance, but overall capabilities remain immature.

The article concludes that the industry needs more real-world benchmarks to drive progress in agentic AI.

Original post →

Benchmark reveals LLMs struggle with real-world tasks despite coding prowess

More from coding & agent