Benchmark Heaven Launches Open-Source Aggregator for 800 Models' Benchmarks and Real Task Costs
On Sept 18, developer airesearch12 launched and open-sourced Benchmark Heaven (beta), a free, ad-free, unfunded AI model evaluation aggregator. It covers 100 benchmarks and 800 models, placing benchmark scores and pricing in a single interactive ranking table, with the core idea of making model comparison reflect real usage costs and behavior.
Confirmed
- Covers 100 benchmarks and 800 models with a unified Composite Score
- Introduces the Benchmaxxing score to measure a model's tendency to game benchmarks; some users cited it to claim DeepSeek's new model "looks like it's benchmaxxing"
- Compares models by real per-task cost, factoring in input/output tokens, inference time, and provider filtering, rather than per-million-token pricing alone
- Users can switch scoring dimensions and views, including Composite, AA Coding Agent, Design Arena Elo, and Epoch ECIE; the site claims "the most detailed cost–capability analysis"
- Offers a data API for building model routers, integrating four data types: model listings, capability scores per topic, real per-task costs, and inference cache prices; the author's Jev auto model router pulls from this API, which the author calls a unique integration
- The project is open source; the author calls it the best data source for model comparison
Why it matters
- The author's central argument is that "per-million-token pricing is a lie": a cheaper model that thinks twice as long may cost more per completed task, offering a new lens for evaluating reasoning models
- The Benchmaxxing score quantifies concerns about benchmark gaming and overfitting into a comparable metric
- The interactive ranking table plus open API (capability scores, task costs, cache prices) enables third parties to build applications such as model routers
- Its free, open-source, ad-free and unfunded positioning offers an auditable alternative to commercial evaluation sites
Timeline
- 09-18: Benchmark Heaven beta launched as open source; the author repeatedly introduced its aggregation, real-cost pricing approach, and API
2026-09-18 ~ 2026-09-19 · 12 related posts
Primary sources
- Benchmark Heaven aggregates 100 benchmarks and 800 models, adds first Benchmaxxing score — airesearch12 ·
- Open-source Benchmark Heaven compares models by cost per actual task, not per-token price — airesearch12 ·
- Benchmark Heaven Launches: 100 Benchmarks, 800 Models, Real Per-Task Costs — airesearch12 ·
- [source] Benchmark Heaven Launches: 100 Benchmarks, 800 Models, Real Per-Task Costs — airesearch12 · 2026-09-18
- [source] Benchmark Heaven aggregates 100 benchmarks and 800 models, adds first Benchmaxxing score — airesearch12 · 2026-09-18
- Price per million tokens is a lie: new benchmark scores models by cost per actual task — airesearch12 · 2026-09-18
- [source] Open-source Benchmark Heaven compares models by cost per actual task, not per-token price — airesearch12 · 2026-09-18
- One API bundles model lists, per-topic capability scores, task costs and cache pricing for building LLM routers — airesearch12 · 2026-09-18
- Benchmark Heaven: A Fine-Grained Cost-vs-Capability Explorer for Every Model and Provider — airesearch12 · 2026-09-18
- Benchmark Heaven adds benchmaxxing signal to spot models gaming famous benchmarks — airesearch12 · 2026-09-18
- Benchmark Heaven flags 'Benchmaxxing': top scorers average on uncontaminated tasks — airesearch12 · 2026-09-19
4 near-duplicate retellings: airesearch12 · airesearch12 · airesearch12 · airesearch12