MiniCPM5-2B Scores 100 vs Spark-X2.5-4B's 84 on a Real Agent Task
Equivalent-Grass-527 · reddit · 2026-09-16
A hands-on comparison tested two small LLMs on a real customer-service agent task with database-backed validation: look up an order, check policy, find inventory, create a replacement, and schedule pickup. The catch: the nearest Shanghai warehouse was out of stock (Suzhou, 95km away, had 12 units), and 'tomorrow afternoon' had to resolve to Sept 10 given the system date.
- MiniCPM5-2B: completed the full chain, chose Suzhou, scheduled pickup for 2026-09-10 14:00–18:00. 100/100, 8.35s, 527 completion tokens.
- Spark-X2.5-4B: got the first four steps right but stopped to ask the user for a time range despite the default being sufficient — replacement created, pickup not scheduled. 84/100, 12.6s, 1,460 tokens.
The author argues the difference lies in the 'last mile of agency' — finishing the workflow vs pushing decisions back to the human — something standard benchmarks miss, and proposes end-to-end task-completion/database-state evals as more meaningful.
More from coding & agent
- Microsoft's MemSteer turns unlabeled user histories into agent memory supervision — MicrosoftResearch · 2026-11-02
- She fired herself from her own Instagram: AI agent runs the account, views up 82.3% — heyshrutimishra · 2026-11-02
- Sourcegraph: AI writes code faster than teams can review it, driving more outages — AI Engineer · 2026-11-02
- Tsinghua & Microsoft's HarnessMM cuts agent token use by up to 87.93% while raising success rate — rohanpaul_ai · 2026-11-02
- GitHub launches Agent HQ: Anthropic, OpenAI, Google coding agents in Copilot — emmanuelvivier · 2026-11-02
- DeepSeek open-sources Super-Agent, a coding agent that reads team chats to minimize human input — emmanuelvivier · 2026-11-02