MiniCPM5-2B Scores 100 vs Spark-X2.5-4B's 84 on a Real Agent Task
Equivalent-Grass-527 · reddit · 2026-09-16
A hands-on comparison tested two small LLMs on a real customer-service agent task with database-backed validation: look up an order, check policy, find inventory, create a replacement, and schedule pickup. The catch: the nearest Shanghai warehouse was out of stock (Suzhou, 95km away, had 12 units), and 'tomorrow afternoon' had to resolve to Sept 10 given the system date.
- MiniCPM5-2B: completed the full chain, chose Suzhou, scheduled pickup for 2026-09-10 14:00–18:00. 100/100, 8.35s, 527 completion tokens.
- Spark-X2.5-4B: got the first four steps right but stopped to ask the user for a time range despite the default being sufficient — replacement created, pickup not scheduled. 84/100, 12.6s, 1,460 tokens.
The author argues the difference lies in the 'last mile of agency' — finishing the workflow vs pushing decisions back to the human — something standard benchmarks miss, and proposes end-to-end task-completion/database-state evals as more meaningful.
More from coding & agent
- Two easy ways to turn an idle Mac mini into a cloud coding agent — iannuttall · 2026-09-16
- HF Jobs OCR script zoo: 28 models, one command, 7 pages for 7 cents — vanstriendaniel · 2026-09-16
- Creator's 3-step system turns 15 manual podcast steps into 8 AI skills, saving 5 hours a week — petergyang · 2026-09-16
- Friends vibe code a personal chat app with retro Windows-style UI, planning a shareable release — floguo · 2026-09-16
- Cumora: open-source team chat where AI agent teams collaborate with humans — tom_doerr · 2026-09-16
- EDSL to add activation capture, probes and model steering for LLM social simulation research — soumitrashukla9 · 2026-09-16