Pareto Frontiers Should Be Convex Hulls: Princeton Team Slams Flawed Model Benchmark Charts
random_walker · x · 2026-09-28
Arvind Narayanan's team points out that popular model comparison charts misuse Pareto frontiers: Ember and GLM are not on the frontier, because a router can switch between Sol and Gemini to hit any cost-accuracy point on the chord connecting them — adding models should only grow the area under the curve, never shrink it.
They made these points two years ago in AI Agents That Matter (arXiv:2407.01502), one of the first papers to call for Pareto curves over accuracy-only tables. Key arguments:
- Accuracy-only focus yields needlessly complex, costly SOTA agents and wrong conclusions about what drives gains
- Joint accuracy-cost optimization can slash cost while preserving accuracy
- Model-developer and downstream benchmarking needs have been conflated
- Weak holdout sets let agents overfit via shortcuts
- Lack of evaluation standardization undermines reproducibility
More from coding & agent
- Personal agents from big labs vs one shared company agent: which design wins? — AshishKulkarni1411 · 2026-11-16
- WebArena-Pro launches: 300-task multimodal web-agent benchmark across 20 apps — xhluca · 2026-11-16
- Self-improving agent papers this week: agents learn the benchmark, not the task — TheTuringPost · 2026-11-16
- 1,000 agents, one week, 630k lines: HF builds a weights-free Python chatbot — _lewtun · 2026-11-16
- One-person content agency targets $30K/mo with 11 AI agents — full playbook — VibeMarketer_ · 2026-11-16
- Hugging Face releases definitive fine-tuning guide for 3D generation model TRELLIS.2 — ariG23498 · 2026-11-16