Benchmark reveals LLMs struggle with real-world tasks despite coding prowess
数字生命卡兹克 · wechat · 2026-08-17
This article analyzes several benchmarks designed to test LLMs on real-world tasks, revealing a significant performance gap compared to their success in coding.
Key Findings
- E-commerce: In AliExpress's RealReplicaBench (107 real tasks), only Claude Opus 5 passed 60.75%, with most models under 50%. Tasks involve complex, messy data like sourcing, logistics, and disputes.
- Product/Lifestyle: Benchmarks like Tencent's E-Bench (Gaming/Music apps) and Xiaohongshu's VibeLifeBench show low average pass rates and poor stability across multiple runs.
- Coding vs. Real Work: Models achieve >70% on coding benchmarks like DeepSWE but often fail real-world tasks. The gap is attributed to the unstructured, chaotic nature of real work.
Evaluation Mechanism
- Benchmarks simulate full workflows (e.g., using tools, checking email states) with strict grading (one error fails the task), emphasizing results over process.
- Specialized agent frameworks (like Accio) improve performance, but overall capabilities remain immature.
The article concludes that the industry needs more real-world benchmarks to drive progress in agentic AI.
Related event: AI models fail real-world work benchmarks as e-commerce agents fall short(2 posts)→
More from coding & agent
- Apple code signing drags for hours; official advice: just wait a week — pjausovec · 2026-10-16
- Hermes: a 4B model with 8K context got worse with more test-time compute, pointing to contextual reasoning — mangahomanga · 2026-10-16
- Flux 3 Klein's Default Templates Are Broken Without Its Prompt Upscaler — StrictMove5839 · 2026-10-16
- 21-person team ships hundreds of PRs weekly with AI writing 97% of the code — rseroter · 2026-10-16
- ClosedMCP: Set Up MCP Servers Once, Share Them Across Your Whole Team — Ok_Sprinkles1301 · 2026-10-16
- Heisentools: let an agent rewrite a mock.py until it mirrors any real tool's API — kieranklaassen · 2026-10-16