Open-source Benchmark Heaven compares models by cost per actual task, not per-token price

airesearch12 · x · 2026-09-18

Developer airesearch12 released Benchmark Heaven, a free open-source project arguing that price-per-million-tokens is a lie: a cheaper model that thinks twice as long isn't actually cheaper. The tool ranks models by cost per actual task, respecting provider filters and accounting for tokens in/out, reasoning, and caching. By its metric, GLM-5.3 is less benchmaxxed than Gemini 3.8 Flash.

Related event: Benchmark Heaven Launches Open-Source Aggregator for 800 Models' Benchmarks and Real Task Costs(12 posts)→

Original post →

More from Models

• 'You Can Feel the Gradient Slopes': Dev Contrasts Benchmaxxed RL Models vs General Ones — theteknosaur · 2026-11-02
• LightOnOCR-3 Launches: Tops FRBench, Adds Layout Structure and Chart Extraction — IgorCarron · 2026-11-02
• NVIDIA's Kumo Is a Family of Four Tabular Foundation Models, Not Just One — pandeyparul · 2026-11-02
• SemiAnalysis: safety classifiers are underrated — Meta's Muse tops cyber benchmark at 35.2% — ThomasScialom · 2026-11-02
• DeepSeek V4.1 Flash 1017 wins over a longtime closed-source-only user: fast, cheap, reliable — _bachrc · 2026-11-02
• Local quants vs Claude on 9 real private bugs: 3-bit costs thinking time, not score — FeydRowan · 2026-11-02

Models channel →