CARTBench from Zhejiang U & Xiaohongshu: no model tops 45% on excellent shopping advice
Yuqin Dai · hf · 2026-11-12
Zhejiang University, Tsinghua, Xiaohongshu and SJTU release CARTBench, a rubric-based benchmark for LLM shopping assistants handling Complex and Ambiguous Requests with Timelines:
- Curated from real commercial shopping needs across 17 retail domains, with timestamped dialogues and 1,603 product records
- 1,328 expert-annotated rubric criteria across Core / Excellence / Safety, distilled from successful orders
- Full-suite eval of 7 flagship API models plus 35B+ open-weight models: only 3 of 10 exceed 90% on Core, and none exceeds 45% on Excellence, revealing an understanding-to-execution gap
- Shared failure modes: weak product evidence and unresolved needs tied to changing requirements and decisive unknowns
Code and data are open-sourced, with a Principle-Guided CART Prompting strategy proposed alongside.
More from Research
- DeepMind Podcast: AlphaGenome 2 Helps Scientists Screen Millions of DNA Changes for Rare Disease Causes — GoogleDeepMind · 2026-11-12
- Test-time self-distillation decoding beats beam search by 8.2 pts on MATH500 without training — hbouammar · 2026-11-12
- Anthropic's 'Worse Together': single coordinator agent beats per-user agent teams in all 4 tests — ombre-ai · 2026-11-12
- 1,440 Synthetic Control Specs Show Huge Leeway in California $20 Minimum Wage Studies — RexDouglass · 2026-11-12
- ICLR abstract word-frequency shift: 'novel' and 'comprehensive' plummet as LLM fingerprints fade — y_m_asano · 2026-11-12
- Evolving Infra: agent-adaptive LLM serving cuts latency up to 4.1x vs NVIDIA Dynamo — mit-han-lab · 2026-11-12