Evaluating One Model on ProgramBench Now Costs Over $10k
jyangballin · x · 2026-10-01
KLieret notes that frontier benchmark evaluation has never been more expensive: evaluating a single model on ProgramBench now costs over $10k. He shares an alternative ranking based on tasks completed per dollar instead of raw scores.
More from Models
- Fireworks launches Ember-1, built on Kimi K3, matching quality with fewer reasoning tokens — sophiamyang · 2026-10-01
- MiniMax Teases MiniMax Code: A Fresh Start, Not Just for Code — iamaliveix · 2026-10-01
- Interleaved Head Attention Accepted at NeurIPS, Boosts RULER Retrieval 10-20% — _arohan_ · 2026-10-01
- Dev praises Opus 5.5 status messages for reporting progress like an expert engineer — sytelus · 2026-10-01
- Untrained d1 model plays Smash Melee at impressive level, researcher stunned — maximelabonne · 2026-10-01
- GPT-6 Astra prefers CDT over EDT, unlike Anthropic models, suggests decision-theory sycophancy gap — dfrsrchtwts · 2026-10-01